MapInfo Pro Developers User Group

 View Only
Expand all | Collapse all

Memory Issues

  • 1.  Memory Issues

    Posted 08-04-2017 21:25
      |   view attached

    This entry may be a little long and with lots of data/code but I am having some serious issues and I am going crazy trying to figure them out.

     

    I am programming in Visual C++, Visual Studio 2013. I am using MapInfo Pro 15.0.2. I am using the MapInfo/MapBasic API calls to Do and Eval to run MapBasic commands. When I try to write to a MapInfo table and commit that table I am getting errors. They are not always the same error and don't always occur at the same "time" during the running of my program. Attached is a file (MIErrors.png) that show some of them. They include:

     

    1) Access Violation in MapInfow.exe

    2) Heap Corruption

    3) The data stops being written to the MI Table , but no actual crash, and I see this as a reason: Out of disk space while editing table nnn_RSSI_CW_LTE_UpperC_22_0_Cha. Please revert your changes to avoid database corruption.

    4) Internal Error: 34623

    5) Internal Error: 34531

     

    Below is the code that I have in my C++ code:

     

    if (!TableExists(mappingInfo->m_cstrMITableName))

    {

    1) cstrCommand.Format("Create Table \"%s\" (%s) File \"%s\" TYPE NATIVE "

    "Charset \"WindowsLatin1\"", mappingInfo->m_cstrMITableName,

    mappingInfo->m_cstrTableQuery, pathMITable.m_strPath);

    retVal = Do(cstrCommand);

    }

     

    if (retVal)

    {

    // Open the tables

    2) mappingInfo->m_cstrMIAlias = OpenTable(pathMITable.m_strPath, "", true);

    // Create the map for the output file

    3) cstrCommand.Format("Create Map For %s", mappingInfo->m_cstrMITableName);

    retVal = Do(cstrCommand);

     

    if (retVal)

    {

    break;

    }

    }

     

    **********************************************************************************

     

    try

    {

    m_pMI->ShowProgressBars(false);

    sCmd.Format("Insert Into %s (Index, RunNumber, X, Y %s, Obj) "

    "Values (%d, %d, %.4f, %.4f %s, CreatePoint(%.4f, "

    "%.4f))", m_LostConnData->m_cstrMIAlias, sCols,

    iIndx, m_iCurrentRunNumber, X, Y, sVals, X, Y);

    4) bWriteMITable = m_pMI->Do(sCmd);

    system_clock::time_point tp = system_clock::now();

    milliseconds t = duration_cast<milliseconds>(tp - m_SaveInterval);

     

    if (t.count() > 3000) // 1000)

    {

    5) m_pMI->CommitTable(m_LostConnData->m_cstrMIAlias);

    m_SaveInterval = tp;

    }

    m_pMI->ShowProgressBars(true);

    }

    catch (CString ex)

    {

    CString erMsg;

    erMsg.Format("Trying to save data to MapInfo File. Error %s "

    "in CMeasurementDevice::RecordMeasurements",

    ex);

    AfxMessageBox(erMsg);

    }

    catch (COleDispatchException* e)

    {

    CString erMsg;

    erMsg.Format("Map Basic Command Error: %s. Error in "

    "CMeasurementDevice::RecordMeasurements",

    e->m_strDescription);

    e->Delete();

    AfxMessageBox(erMsg);

    }

    catch (...)

    {

    AfxMessageBox("Trying to save data to MapInfo File. Unknown Error in "

    "CMeasurementDevice::RecordMeasurements");

    }

    1) An example of my create table command is:

    Create Table "test_RSSI_CW_LTE_UpperC_22_0_Chan_5220_RSSI" (Index Integer, RunNumber Integer, X Float, Y Float, Max_RSSI FLOAT, Max_Chan INTEGER, Chan_5220_RSSI FLOAT) File "D:\Application Testing\Spectra_Testing\test_RSSI_CW_LTE_UpperC_22_0_Chan_5220_RSSI.TAB" TYPE NATIVE Charset "WindowsLatin1"

    2) Open table is equally as simple and straight forward:

    Open Table "D:\Application Testing\Spectra_Testing\test_RSSI_CW_LTE_UpperC_22_0_Chan_5220_RSSI.TAB" Interactive

    3) Finally I Create a map for the table:

    Create Map For test_RSSI_CW_LTE_UpperC_22_0_Chan_5220_RSSI

    In a piece of code not shown here, I do a simple: "Map From test_RSSI_CW_LTE_UpperC_22_0_Chan_5220_RSSI". Overall, these three commands are good and as I said simple.

    4)This is the insert command I run. An example is:

    Insert Into test_RSSI_CW_LTE_UpperC_22_0_Ch (Index, RunNumber, X, Y , Chan_5220_RSSI, Max_Chan, Max_RSSI, Obj) Values (1, 496, 51.9703, -73.2853 , -116.569999694824, 5220, -116.57, CreatePoint(51.9703, -73.2853))

    This is again a very simple insert with the creation of a point object included. This "works". That is it writes the data to the file for several loops until there is one of the errors. When this Insert stops and I get the "Out of disk space ...." error it is very confusing.

    5) The Commit Table command is also straight forward:

    Commit Table "test_RSSI_CW_LTE_UpperC_22_0_Chan_5220_RSSI"

    I have been running with and without this statement. The prescence of the statement changes where and when and sometimes what error I get. If it is in there, I tend to get an Access Violation on the very first attempt at a commit. With out it I get several data lines written to the MI table before I get an error.

    What is frustrating is that these are all normal MapInfo/MapBasic commands and commands issued without problem in other areas of our program. We used to have a Linked, then a Live table that was linked to an Access database but were having trouble with that. So I decided to try and control writing to both the Access and MI table. This code is my attempt at that and therefore fairly new code. When the MI table was linked (Live or Linked) to the Access table, the updates to the MI Table were good (no errors) but they lagged. For example, when driving around and plotting data we would get to where the data being shown was several (10-20) minutes behind where the car actually was. So to try and fix this I did as I said. I am trying to write to the Access DB and then at the same time, the MI Table. Unfortunately I am getting these errors.

     

    Errors that all point to some time of memory issue. But I cannot see what I am doing wrong or where I am having a memory issue. There is also another issue that I cannot fully explain here as it is a visual problem, but in the attached file is a screen shot of what our program looks like when displaying data (albeit an incomplete picture). The black dots that are data at X,Y points should be fully spread from Green to Red pin. They never get fully from Green to Red as I get the error(s) I have talked about. But also, sometimes, the dots are printing then the screen "flickers" and some of them are gone, overwritten by a white block, erased, or I don't know what.

     

    Any help, ideas, tests, what ever anyone can think of to help me would be greatly appreciated.



  • 2.  RE: Memory Issues

    Posted 08-08-2017 09:31

    Hi Bill,

    Working with our MapInfo Pro Engineering team for further investigation. Thanks, - Matt



  • 3.  RE: Memory Issues

    Employee
    Posted 08-09-2017 03:07

    Bill,

    I have no good idea why you are seeing these errors as this really is very simple process.

    You said that you were writing to the MapInfo Table as well as to an Access database and that you earlier just were writing to the linked MapInfo table where it worked but it was lagging (a lot).

    Did you try only inserting the data into your Access table and then - once in a while - do a Server Refresh on the linked table in MapInfo Pro?



  • 4.  RE: Memory Issues

    Posted 08-09-2017 08:13

    Peter,

    Thank you for responding. The option you describe, running Server Refresh is indeed what we were doing. I don't know of another way to update an MI table with a linked Access DB file. If you do know of another way to do that, please let me know and maybe I can try it. But we would record a few dozen records to Access then every 3 seconds do a Server Refresh.

    Thanks,

    Bill



  • 5.  RE: Memory Issues

    Employee
    Posted 08-09-2017 08:32

    Bill,

    The other would be to insert the data into the linked table in MapInfo Pro, save the changes and potentially also do a refresh. The refresh would only be necessary if you know that things might also have changed with the data in MS Access.

    From what I understand from your post, the process of inserting data into Access and refreshing the linked table, caused the data to lag, right? It was showing data 20 minutes old?



  • 6.  RE: Memory Issues

    Posted 08-09-2017 08:44

    Peter,

    I actually did try the direct write to the MI table while linked and couldn't get it right. I don't know if I missed something or had a wrong instruction, but it didn't go smoothly. Maybe I should try again.

    You are correct, the data is lagging 10 to 20 minutes behind for a 5 minute drive test. We would return from the drive and have to wait to see all the data points appear on the mapping window along route we took.

    Bill



  • 7.  RE: Memory Issues

    Employee
    Posted 08-09-2017 08:55

    Let me understand something in your setup:

    1. You have this program running on a laptop that you bring with you in a car which will connect to "some other device" and collect information and coordinates while driving.
    2. Your tool inserts the collected data into either a. a linked table, b. a native table or c. an Access database
    3. For a: This didn't work - maybe you'll run into the same issues as with b ?! I don't know
    4. For b: you get all sorts of errors as reported above
    5. For c: data was lagging

     

    I wonder why the data was lagging so much during a short five hour test drive. Can you see the data in the Access database? Could it be that the refresh process takes more time than you expect and that this is causing the lag?

    I'm kind of puzzled by your riddle here :-)

     



  • 8.  RE: Memory Issues

    Posted 08-09-2017 09:02

    Peter,

    Yes, you do have the situation correct. For a, as I said, I may have done something wrong and can retry again sometime soon. I am currently trying to get b working at this time (have ticket with PB and just got a response from them). c was indeed lagging, and as the same with you,

    From testing it is indeed the refresh that is lagging. I added some timing code and the time from data received from the RF scanner to successful write in the Access DB was less than 1/100000 of a second (it barely took any CPU "clicks" to update.) The lag was definitely around the server refresh.

    Sorry to puzzle you, but PB support is also very puzzled. Again, thank you for at least trying to figure this out.

    Bill



  • 9.  RE: Memory Issues

    Posted 08-09-2017 10:43

    UPDATE: I got the following email from PB support and have included my response to them.

    **************************************************************************************************************

    Hi Bill,

    I let the engineering know that the way your code works is you Call MapInfo Pro, throught VS first and then manually click to run the MBX second to create the points..

     

    Engineering asks the following: 

    1. Are you running from within VisualStudio when the exception occurs? If so, can you [Break] from the dialog and then capture the stack trace (select the Call Stack window; cntl-a/cntl-c to select and copy all levels).

    2. If you are not running from within VS, I’m not sure why Pro cannot close in order to generate the dump file we are looking for. If it’s not actually crashing, then it must be waiting on something. 

    (we created an environmental variable ) MI_FULLDUMP so we can get that dump in the %temp% directory but we did not have success because MapInfo Pro could not crash/close by itself. 

    *****************************************************************************************************************

    Matt and Engineering,

    I answer the questions and update you below, but as I was generating the data for this email, I started to wonder. Is it possible that my installation of MapInfo is “corrupt”? Now Matt and I went and updated the install, which said it was a good install, but I am questioning that. Can I get the v15 installation package sent to me so I can reinstall? I believe I can get it off the web, but I thought I would try a different route to the install package and ask you.  Also, there is the fact that the Internal Error was showing up in MI 12.5 before we upgraded to MI 15.0. So there are three possibilities:

    1. A reinstall of V15 will fix the issues.
    2. My assessment that I may have a corrupt install is false and reinstallation will not change anything.
    3. There something, like a file or a setting that would survive an uninstall of 12.5. If that is so, then maybe it was bad/corrupt and when 15 was installed it simply used this bad “thing” and continues to show me problems.

     Any thoughts or ideas on this?

    The questions below sound as if Engineering is looking for information on the Internal Errors only. That is fine, but I do want to make sure so I don’t send “false” information for the other issue(s) (Access violation errors, Heap corruption).

    The errors are only showing up in Debug mode at this time as I am not able to produce a release version of our code. Therefore the errors are indeed only showing up when running through Visual Studio. 

    The Internal Error is very rare since I have gone to v15. As a matter of fact, I have only seen it twice. Therefore it has proven to be elusive and unable to reproduce and I cannot capture it’s call stack at this time. If it re-occurs, I will post/send it along.

    Regardless of my question above about which error and lack of an Internal Error stack trace, I decided I would do the stack trace for the Access Violation also. So that is here, with a screen shot of the actual error: (Note: This is only one of the “types” of Access Violations I get. There are others with a location that is not 0x0000000)

    >            micore.dll!104249c5()  

                   [Frames below may be incorrect and/or missing, no symbols loaded for micore.dll]           

                   micore.dll!103d0647() 

                   micore.dll!10290006() 

                   micore.dll!103d0033() 

                   micore.dll!101cf0c4()   

                   micore.dll!101cf56b()   

                   micore.dll!10277a9a()  

    Here is a different stack trace from another Access Violation on 0x0000000

                   micore.dll!1027cfc7()   

                   [Frames below may be incorrect and/or missing, no symbols loaded for micore.dll]           

                   micore.dll!103d001f()  

                   micore.dll!101cf0c4()   

                   micore.dll!101cf56b()   

                   micore.dll!1027be41() 

                   tmmon.dll!7440168d() 

                   kernel32.dll!_HeapFree@12() + 0x14 bytes        

    >            msvcr120.dll!free(void * pBlock) Line 51              C

                   MapInfo.Utility.dll!0f65813f()   

                   micore.dll!10169d67() 

                   micore.dll!10276fc7()   

                   micore.dll!10277a2e()  



  • 10.  RE: Memory Issues

    Posted 08-09-2017 10:44

     

     

     

     

    Second part of the response

    *******************************************************************************

    The above traces were run without the commit table call done in 3 second intervals after the Insert Into statement (see code supplied with original entry of ticket, but also reproduced at the very end of this correspondence). Below is the trace with that commit table active. It is still showing a violation on memory 0x000000

                   milib.dll!004c6916()       

                   [Frames below may be incorrect and/or missing, no symbols loaded for milib.dll]               

                   milib.dll!004c66d5()       

                   milib.dll!004ef643()       

                   milib.dll!004d3acb()       

                   tmmon.dll!7440158d()  

    >            msvcr120.dll!malloc(unsigned int size) Line 92 + 0x3b bytes       C

                   micore.dll!101e0ea3()   

                   micore.dll!10207022()  

                   micore.dll!101e65d5()  

                   micore.dll!10216d5d()  

                   tmmon.dll!7440158d()  

                   tmmon.dll!7440168d()  

                   KernelBase.dll!_FindClose@4() + 0x93 bytes      

                   tmmon.dll!7440158d()  

                   tmmon.dll!7440168d()  

                   kernel32.dll!_HeapFree@12() + 0x14 bytes        

                   msvcr120.dll!free(void * pBlock) Line 51              C

                   micore.dll!101e0f36()    

                   micore.dll!101e101a()   

                   micore.dll!101165e0()   

                   micore.dll!101158d6()  

                   micore.dll!1011579f()   

                   micore.dll!10278998()  

                   micore.dll!100bb4cc()   

                   tmmon.dll!7440158d()  

                   tmmon.dll!7440168d()  

                   tmmon.dll!7440168d()  

                   kernel32.dll!_HeapFree@12() + 0x14 bytes        

                   msvcr120.dll!_unlock(int locknum) Line 368       C

                   msvcr120.dll!malloc(unsigned int size) Line 92 + 0x3b bytes       C

                   oleaut32.dll!_DispCallFunc@32() + 0xa6 bytes  

                   mis2ops.dll!006d006d()                

    /////////////////////////////////////////////////////////////////////////////////////////////////////////////////

    m_pMI->ShowProgressBars(false); 

    sCmd.Format("Insert Into %s (Index, RunNumber, X, Y %s, Obj) " 

    "Values (%d, %d, %.4f, %.4f %s, CreatePoint(%.4f, " 

    "%.4f))", m_LostConnData->m_cstrMIAlias, sCols, 

    iIndx, m_iCurrentRunNumber, X, Y, sVals, X, Y);

    bWriteMITable = m_pMI->Do(sCmd); 

    system_clock::time_point tp = system_clock::now(); 

    milliseconds t = duration_cast<milliseconds>(tp - m_SaveInterval); 

    if (t.count() > 3000) // 1000) 

    m_pMI->CommitTable(m_LostConnData->m_cstrMIAlias); 

    m_SaveInterval = tp; 



  • 11.  RE: Memory Issues

    Posted 08-09-2017 12:35

    Got a new Internal Error message: 34621. Here is the stack trace:

     

    >            user32.dll!_NtUserGetThreadState@4() + 0x15 bytes    

                  user32.dll!_GetMessageTime@0() + 0x7 bytes  

                  micore.dll!1015b76c()  

                   [Frames below may be incorrect and/or missing, no symbols loaded for micore.dll]           

                  micore.dll!101650f8()  

                   milib.dll!0027527d()     

                   milib.dll!002760cd()      

                   milib.dll!00279afe()       

                   ntdll.dll!_RtlFreeUnicodeString@4() + 0x539 bytes         

                  KernelBase.dll!_GetModuleHandleA@4() + 0x40 bytes 

                  005c4ae3()         

                  MapInfow.exe!00401016()         

                   MapInfow.exe!0040124f()          

                   kernel32.dll!@BaseThreadInitThunk@12() + 0x12 bytes

                  ntdll.dll!___RtlUserThreadStart@8() + 0x27 bytes           

                  ntdll.dll!__RtlUserThreadStart@8() + 0x1b bytes