locked
PocketSense scrubber discussion for those who download transactions with PocketSense already RRS feed

  • General discussion

  • PocketSense, lets people using Microsoft Money Plus download transactions from many US brokers and credit cards.  While it can be used as a way for any Money users to get quotes, the best use is for getting transactions via OFX servers. This thread is intended for those who use Pocketsense for transaction downloading already, but want to implement a modification to the scrubber.py file to modify the downloaded data. The majority of PocketSense users don't need to even be aware of scrubbers.

    I have made scrubber modifications for my own use. I wanted to open discussion on what scrubber operation people have made  for themselves or for those who consider it.




    Saturday, July 14, 2018 9:01 PM
    Moderator

All replies

  • I have made a demo of a scrubber for those who might want to experiment.

    sc_x2.zip is temporarily available at  http://s000.tinyupload.com/?file_id=07871396969749411297

    The SHA1 for the zip file is cf5ccc6ac784faa03e3c4d942f0aab6f490992d6 for those few who might care.

    The zip file includes a text file that includes this content:

    sc180528.py is a copy of the Pocketsense 28-May-2018 scrubber

    sc_x2.py is a modified version of that original scrubber.py intended for scrubber experimenting
    and learning. It adds scrubbers for Fidelity and Schwab. The modified file must be named scrubber.py and replace the existing scrubber.py.  The modifications  do nothing
    different from the original scrubber.py except print to the screen including
    the account number processed. This is just to demonstrate that the broker-specific scrubber routine is being invoked and is accessing the ofx file.

    You can use the windows FC command to see the differences in the two files.

    These files are for learning and experimentation for people who download transactions with PocketSense and might have a special need. If you get that working as far as printing the account number, you can ask back to see if somebody could tell  you how to do some actual thing that could be useful.

    For example, somebody might make a scrubber that includes a list of <UNIQUEID> values for which the scrubber should remove the <UNITPRICE> info. That seems like an unlikely thing to want, but it could fit a desire that I have seen discussed recently.





    Sunday, July 15, 2018 7:32 PM
    Moderator
  • Thank you Cal.  I'll keep the file for reference, but along with <UNITPRICE>, <UNITS> and <MKTVAL> are also reported for each CD, and as Robert alluded to, the Money account balance comparison would get messed up just changing <UNITPRICE>, so I might not bother.  Besides, I don't know that I could write the python script for Kevin.  I was going to replace the <UNITPRICE> values with 100.0000

    Schwab Data:

    <POSDEBT>
    
                <INVPOS>
    
                  <SECID>
    
                    <UNIQUEID>381426PE9
    
                    <UNIQUEIDTYPE>CUSIP
    
                  </SECID>
    
                  <HELDINACCT>OTHER
    
                  <POSTYPE>LONG
    
                  <UNITS>????
    
                  <UNITPRICE>100.6956
    
                  <MKTVAL>????
    
                  <DTPRICEASOF>20180722210000
    
                  <MEMO>GOLDMAN SACHS BA 5.25%18CD FDIC INS DUE 10/09/18US
    
                </INVPOS>
    
    </POSDEBT>

    x = re.search("<UNIQUEID>(\w*)\W",ofx) #finding uniqueID
        an if statement to compare to string of unique ID groups: 381426PE9; YYYY; ZZZZ  OR in my case if FDIC is found in the <MEMO> field

        if matched, alter the <UNITPRICE> value for each group within <INVPOS> and </INVPOS>  

    I don't know if Fidelity uses <INVPOS> to differentiate the CD groups.





    • Edited by ameridan Monday, July 23, 2018 5:20 PM
    Monday, July 23, 2018 4:48 PM
  • I could write the python for the scrubber routine, but only if there was actual interest. Being able to implement the do-nothing-but-print-to-the-screen would demonstrate the interest and the ability to implement.

    I always thought of the scrubber.py as being something intended for advanced user use. I have made some. It would be nice if the distributed scrubber.py would be written to make it easier to keep mods with a new version package. As it is, you would have to be careful to not overwrite your Schwab etc scrubber. It's easy enough to do. In a re-think of scrubber, there could be an entry in a sites descriptor for a scrubber routine. That way, you would not worry about accidently overwriting your Schwab scrubber (for example).

    An alternative to writing a unit price of 100, it seemed to me that you could just eliminate UNITPRICE in the area where quotes are. You would not want to modify UNITPRICE for an actual transaction, so care would be needed there.

    So anyway, if MEMO containing FDIC as a word is a sufficient identifier, that would be an easier check than CUSIP. CUSIP would be more universal, but I don't think universal is important in this case. There is very limited interest in the problem.

    Regarding the question marks in your segment, I presume you would just leave UNITS unchanged.  For the MKTVAL, maybe eliminate that field, unless experiment shows it to be needed. If not having that causes it to have an error, then put in a 100 * units calculation result.

    You might experiment to see what you can just be rid off in the INVPOS record.



    Monday, July 23, 2018 5:13 PM
    Moderator
  • Again, thank you Cal.  We'll just leave this one alone.  My question marks simply replaced actual data.  Robert's been good about incorporating scrubber routines from others into his updates, but you are correct; if specialized ones were developed, they could get wiped out in updates.  Robert would probably fold in a pointer to also run a supplement scrubber2.py (if exists) subsequent to the primary scrubber.py, that would be user supplied/customized and wouldn't be an included file in updates, if requested ;)



    • Edited by ameridan Monday, July 23, 2018 6:06 PM scrubber solution
    Monday, July 23, 2018 5:29 PM
  • MKTVAL is a required field in OFX. In the OFX Spec, a bold item is required, regular text is optional.

    Tag

    Description

    <INVPOS>

    General-position aggregate

       <SECID>

    Security identifier

       </SECID>

     

       <HELDINACCT>

    Sub-account type CASH, MARGIN, SHORT, OTHER

       <POSTYPE>

    SHORT = Writer for options, Short for all others.

    LONG = Holder for options, Long for all others.

       <UNITS>

    For stocks, MFs, other, number of shares held. Bonds = face value. Options = number of contracts quantity

       <UNITPRICE>

    For stocks, MFs, other, price per share. Bonds = percentage of par Option = premium per share of underlying security unitprice

       <MKTVAL>

    Market value of this position, amount

       <DTPRICEASOF>

    Date and time of unit price and market value. Can be 0 if unit price and market value are unknown, datetime

      <CURRENCY>

    Currency information if different from default currency.

      </CURRENCY>

     

       <MEMO>

    Comment, memo

    </INVPOS>

     


    Bob - MSN Mobile

    Monday, July 23, 2018 7:54 PM
  • Thanks Bob, but this is in regards to Kevin's request in the Pocketsense forum, where he simply didn't want the quotes for secondary CDs entered into Money, meaning in essence, just skip the securities updates for those investments.  He states "I'll just delete the unit prices as I perform the import. There's only 6 CDs involved so it's no biggy.", so he's manually deleting this data out of the ofx files anyways, before importing into Money.

    That's why Cal's idea "You might experiment to see what you can just be rid off in the INVPOS record." actually makes more sense, in-line with your comment.

    Monday, July 23, 2018 8:49 PM
  • Bob read me right, but omitting the whole INVPOS for those pieces with memo FDIC could work.

    However just adjusting the price to 100 and changing the MKTVAL works, and lets Money look for any disagreements in the holding.

    Monday, July 23, 2018 9:23 PM
    Moderator
  • On http://pocketsense.blogspot.com/2018/08/new-version-available.html#comment-form neider asked about the potential for a PocketSense scrubber for a specific problem. I wrote some code that could be integrated into scrubber.py.

    The purpose is to translate specific CUSIPs or tickers to whatever you would prefer them to be. The OFX is modified by the scrubber before it is passed to Microsoft Money by PocketSense

    The revised code is http://s000.tinyupload.com/?file_id=01101935077756084112

    SHA1 is  4546f914c05a89f7dbe396ba993b6c00ef685cfd

    It is not as polished as it could be, but it works according to my testing. You would replace VoyaPlans with your broker name....





    Saturday, July 13, 2019 5:40 PM
    Moderator