Speeding up NinjaTrader with a RAM drive (ramdisk)
Welcome to Big Mike's Trading Forum.
Welcome, Guest!
This forum was established to help traders by openly sharing indicators, strategies, methods, trading journals and discussing the psychology of trading.
BMT is fundamentally different than most other trading forums:
We work extremely hard to keep things positive on our forums.
We do not tolerate rude behavior, trolling, or vendor advertising in posts.
We firmly believe in openness and encourage sharing. The holy grail is within you, it is not something tangible you can download.
We expect our members to participate and become a part of the community. Help yourself by helping others.
You'll need to register in order to view the content of the threads and start contributing to our community. It's free and simple, and we will never resell your private information.
Well, that is a good question, but unfortunately somewhat rhetorical, as it is hard to know what NT is doing under the hood, or what windows is doing for that matter. I just know the RAMDrive helps.
Interestingly, Pawnbroker noticed that putting the Gomfolder on the ramdrive did not help. Purely speculation, but I think it is possible that our favorite Parisian puts a little more time into thinking about performance than other coders.
And to further speculate, I think one problem may be the popular yet dangerous idea that "abstracter is better." I know I found myself pissing up a rope trying implement a SQL back end for gomi data. ...But it seemed so beautiful on paper...
Coincidentally, I ran in a a Yogi Berra quote yesterday. "In theory, there is no difference between theory and practice, but in practice, there is."
WHOA! Mine is twenty-someodd MEGA bytes. I just did a recent reinstall of NT, but even my old Ninja folder was only about 150 MB.
My understanding is the DB folder is analogous to a browser cache, in that if something is missing then it tries to get it from the datafeed. I would check with Ninja, or at least do a good backup, but you may be able to clear a lot of that out.
But to answer your question, no it will not fit. There is no compression on the RAMDrive or the image backup it does. (Maybe with Primo but not the one I used)
Tying up a ton of memory would seem like an expensive last resort to me. I'd try emptying the folders (after backup of course) and see if everything still works. If so, then you should be fine with a smaller RAMDrive.
And please share your results. I'm curious to know what my options will be when my RAMDrive fills up too.
-Bob
The following user says Thank You to bob7123 for this post:
I have found the resource hog that is slowing down NT; it's the emini-watch.com indicators _Better_Momentum and _Better_Momemntum2 (latest versions- Better_Indicators_Ninja_25Aug12_jsy).
Here are some times, starting with just a candle chart and changing between 3 to 5 minutes charts.
Candles only = instant.
with Better_Momentum = 9s
with Better_Momentum2 = 9s
with both indicators = 16s
These tests were done with and without a RAM cache and the times were the same. NT was not recording data at the time, because the markets were closed.
Note that NT and my PC run OK whith these indicators, but changing settings takes a long time. There is not much I can do about this directly as the code is closed source, but I can contact the emini-watch and notify the author that he needs a "better" coder.
Size of my NT database
My NT db directory is 218 MB, so it easy for me to cache all of that in a RAM drive.
Notes on Caching SSD's
There are a number of sites on the WWW having recommendations on settings to optimise SSD's. It's common to state that you should disable prefetch and superfetch [e.g. 1 & 2].
"The purpose of these is to pre-load the programs you load from slow hard drive to fast memory (cache) in case you want to run them. With your SSD, there is no need. Disable them and free up some memory and resources and stop a lot of writes to the SSD. However, this may hurt performance in older first gen SSDs and HDDs".
Disabling prefetch makes sense to increase the life of your SSD, but there is debate about disabling superfetch. Read the article on that URL and the comments. If you do turn off superfetch you will disable caching and that may slow down NT (if you don't use a RAM drive).
RAM Disk benefits
NT was no faster with the NT database on a RAM drive when I tested today, with the markets closed. However, it was much faster when I tested in the week with the markets open. More testing is needed to check if my fists tests were accurate, but the activity from processing new data would slow things down in principle.
RAM Disk alternatives
I installed SuperCache 5 today, which can be used to add a memory cache for a disk or to mirror an entire disk in memory. No improvement was seen in NT's speed, but note the comments in the RAM disk benefits section, as the tests were done with the markets closed.
There is a competing comment, FancyCache, which is in beta testing and free to use for 180 days. The product is stable as far as I can tell from forum posts.
I will retest in the week, when NT has market data is coming in.
Both supercache and Fancy cache can optionally cache writes to disk. I strongly recommend not using that option, because you will loose data and risk system corruption if your machine crashes or shuts down before data is written from memory to disk. Also, NT will benefit much more by having ultra fast read speeds and I would expect very little benefit to NT from caching writes.
It stores all chart data. Mine is many GB in size because I have years of tick data.
After you reach your conclusions in the thread, I encourage you to post in the NT8 request thread. This is something that NT should be doing under the hood in order to improve speed, caching these files in memory as needed.
Mike
Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.
Need help? 1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first. 2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses. 3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make. 4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance. 5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers. 6) Help using the forum? Watch this video to learn general tips on using the site.
If you want to support our community, become an Elite Member.
The following 2 users say Thank You to Big Mike for this post:
Favorite Instrument: Major player in Simulated Trading of Globex
Posts: 1,196 since Jul 2009
Thanks: 1,148 given,
2,170
received
NT Database Size, Better Momentum Indicator
Maybe instead of the entire db folder, only recent vintage datastore folders need to placed on the RAMDisk.
By moving old data files to an archive outside of the db folder I reduced the folder size to 500 MB so now would be able to use the RAMDisk to contain the entire db folder, or just the data files. I would assume that the database engine gets loaded into RAM anyway and does not need to be on the RAMDisk.
I am already using Windows ReadyBoost, but have not tried to measure any improvement that it might be giving.
Meanwhile I don't think you cant go wrong by putting in as much RAM as possible, as Mike suggests in next post, and putting any spare memory sticks to work as Ready Boost caches.
The Better Momentum Indicator is said to be a momentum oscillator applied to buy sell volume. There are a few different ways to do this and the slow loading time of tbe emini-watch version could probably be improved:
Take the Momentum of GomCD. Fast loading. You can use BidAsk buy sell volume if you have Gom data, but even if you don't have Gom data, you can use UpDownTick data from the Ninja Tick files.
For real time data use momentum of bid ask volume from OnMarketDepth and for historical data use Momentum of On Balance Volume or Momentum of UpDownTick Volume of a secondary data series. If the secondary data series has a high resolution, such as 1 tick, the indicator will be very slow to load, especially if days back is more than one.
For many days back the most practical way would be to either use OBV or to use a moderately coarse secondary data series, such as around 2000 tick for the ES.
After plotting the oscillator curve you would have to add the features that show the signals.
If anyone is interested in pursuing this we will have to create an Elite thread since Gom indicators fall under that category,
Last edited by Zondor; October 28th, 2012 at 07:03 PM.
The following user says Thank You to Zondor for this post:
@Zondor, please keep in mind this is a non-Elite thread so do not post Elite indicators (Gom*)
Mike
Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.
Need help? 1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first. 2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses. 3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make. 4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance. 5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers. 6) Help using the forum? Watch this video to learn general tips on using the site.
If you want to support our community, become an Elite Member.
The following 2 users say Thank You to Big Mike for this post:
A comparison of the peak heights of GomMomentum swings gives a very similar assessment of the market conditions and the strength of the buying and selling (or lack thereof). That is not a clone of his work, so it does not cause me any moral problems to mention that!
Favorite Instrument: Major player in Simulated Trading of Globex
Posts: 1,196 since Jul 2009
Thanks: 1,148 given,
2,170
received
DB Data Folders Moved to RAMDisk, working fine so far
My PC is now running a 1.5 GB RAMDisk containing the cache, data, day, tick and minute folders moved from the db folder, where there are now junctions linking to them. My workspaces do seem to be loading quite a bit faster, although I have not taken any measurements.
I don't think it's a serious problem if the data in these folders gets lost. They would then be automatically rebuilt from server supplied historical data. I am only backing up the RAMDisk about once per hour, and on computer shutdown.
I am assuming that the folder called NinjaTrader that contains the SQL Server CE Database does not include any data files, so there would be nothing gained by moving it to the RAMDisk.
ROFL! I guess it is "Better" to pay for a proprietary indicator that breaks your system than pay to become an elite member and get indicators that are open and have a community of people who will give you their unbiased opinions of their real value.
My NinjaTrader folder is on D, which is not SSD.
I use this tool to set up my SSD: SSD Tweaker by Elpamsoft.com
I'll stay out of which parameter settings are best, but this tool makes it easy to set them from one convenient panel.
It also allows me to initiate a TRIM cleanup and ensure I the OS is using AHCI drivers. (Which are a lot faster than older ones.)
The following user says Thank You to bob7123 for this post:
How did you come by this data? Or more to the point, what is the quality? Did you just keep NT running on a server someplace, or did you pony up for tick data? Can any of it be converted to binary Gom files?
In thinking about how the backfill application could be extended, I think it would be nice to have a cataloging database to know what I have and if there are any gaps in it.
Well, that's the thing. I cut out a comment I thought was a bit snarky in my previous post, but I guess remains relevant.
I was thinking that all the nice new transistors we get courtesy of Moore's law are wasted by increasingly bloated software. I'll be happy to tell the NT folks to write cleaner code, but I doubt there will be much beyond lip service; a lot of developers these days just figure you can always throw more hardware at the problem.
