Big Mike's Trading Forum
 

Go Back   Big Mike's Trading Forum

> Trading Forums > Platforms and Indicators > NinjaTrader


Ninja Tunes
Started:August 26th, 2013 (11:30 AM) by DionysusToast Views / Replies:2,217 / 9
Last Reply:December 30th, 2013 (08:03 PM) Attachments:0

Welcome to Big Mike's Trading Forum.

Welcome, Guest!

This forum was established to help traders by openly sharing indicators, strategies, methods, trading journals and discussing the psychology of trading.

BMT is fundamentally different than most other trading forums:
  • We work extremely hard to keep things positive on our forums.
  • We do not tolerate rude behavior, trolling, or vendor advertising in posts.
  • We firmly believe in openness and encourage sharing. The holy grail is within you, it is not something tangible you can download.
  • We expect our members to participate and become a part of the community. Help yourself by helping others.


You'll need to register in order to view the content of the threads and start contributing to our community. It's free and simple, and we will never resell your private information.

-- Big Mike
     

Reply
 
Thread Tools Search this Thread

Ninja Tunes

Old August 26th, 2013, 11:30 AM   #1 (permalink)
Elite Member
Bangkok
 
Trading Experience: Intermediate
Platform: S5 Trader, NinjaTrader
Favorite Instrument: ES
 
DionysusToast's Avatar
 
Posts: 1,699 since Nov 2010
Thanks: 505 given, 3,867 received

Ninja Tunes

Following on from some other conversations on Ninja, I'm going to go through some less obvious issues your Ninja might be running slowly....

Issues with Ninja are usually either through thrashing the processor or the disk. Windows performance monitor will tell you which...

Most of the issues I have had with Ninja have been with disk thrashing, so we'll start there.

Disk Thrashing

Ninja stores data on the disk and to some extent, the amount of disk writes depends on the settings & indicators.

1 - Log file/Trace files are written to infrequently
2 - GOMI tools write to a tick store in MY DOCUMENTS
3 - Ninja will write replay data if "Record for market replay" is set
4 - Ninja will write bar data if "save chart data as historical" is set
5 - Ninja writes to an SQL Server Compact database each time you place an order/order is filled

So, there's plenty of writes to the disk. I'd advise NOT saving data for market replay.
I would also advise moving your MY Documents folder to a different physical drive to your system folder (C:\). I do mean physcal drives and not just a partition.

With this done, Ninja is still going to be hitting your disk. Most antivirus programs will inspect a file when it's changed. So if you Gomi on, then your antivirus will scan that file every time a tick comes in. The other program that does this is "SearchIndexer.exe" which populates the database used when you tell windows explorer to search for a file.

What can happen is that each time a tick/bar comes in - the antivirus and searchindexer start working and the hard disk can't keep up. So next time Ninja needs to write a tick, it has to wait for the disk to become available. This is going to slow things down and in the windows resource monitor, the processor utilization will be quite low simply because disk has become the bottleneck and not the processor.

Disabling Search Indexer
Basically, all you have to do is to disable the service:
Windows 8: Disable Search Indexing
Disable Search Indexing in Windows 8

Configuring Anti Virus
I'm no expert here but I can tell you what I did. I set it to exclude certain files and certain processes from virus check

Excluded Files and locations:
Please register to view the post attachment(s), image(s), or screenshot(s) - it's simple and free.

The first is the NinjaTrader 7 folder in My Documents, that's where Ninja keeps it's data/database
The second is the files stored by the GOMI tools

Excluded Processes
Please register to view the post attachment(s), image(s), or screenshot(s) - it's simple and free.

This is the IQFeed client and NinjaTrader process

Note that this is windows defender. Other security programs will have something similar to this to prevent files/processes from being scanned.

NinjaTrader Database
On top of this, you also need to clear out the Ninjatrader trade database. You will notice that as you do more trades, there seems to be a growing pause between clicking to enter a trade and the trade being confirmed. This is Ninja writing to it's database. It gets slower the more times it writes. To clear this:

NinjaTrader Control Center -> Tools -> Options -> Data -> Reset DB

If you have any questions about the products or services provided, please send me a Private Message or use the BMT "Ask Me Anything" thread

Last edited by DionysusToast; August 26th, 2013 at 11:36 AM.
Reply With Quote
     
The following 20 users say Thank You to DionysusToast for this post:
     

Old August 26th, 2013, 11:34 AM   #2 (permalink)
Elite Member
Bangkok
 
Trading Experience: Intermediate
Platform: S5 Trader, NinjaTrader
Favorite Instrument: ES
 
DionysusToast's Avatar
 
Posts: 1,699 since Nov 2010
Thanks: 505 given, 3,867 received

Processor Thrashing - When very little is running

Every now & again, someone comes along with a super fast PC and NinjaTrader hogging one of the CPUs.

Now - just to clarify one thing. If you have an i7 processor, there are 8 cores. NinjaTrader is single-threaded so it runs on only one of those cores. When you look at the NinjaTrader process in task manager it has CPU utilization. This will NOT be 100% when Ninja is hogging a core. It will be 12.5%...

8 cores = 100%, 1 core = 12.5%
or on an i5
4 cores = 100%m, core = 25%.

So on an i7, if you see Ninja using 12.5% it is thrashing the processor, on an i5, if you see Ninja using 25%, it is thrashing a processor.

Now - Ron just had a new machine built and had the same issue. Just running Ninja & Jigsaw and it's pegging a core at 12.5%. This should be impossible - the fact is that you should overload your internet connection before the processor. Or to put it another way, your CPU has the ability to process data a lot faster than the internet can get it to you. Of course, if our software has a bug in it OR you are running 1000 indicators, sure you can overload the CPU. In absence of this sort of issue, even when the market goes crazy, you shouldn't see much more than 2-3% on the core of an i7.

Today, it appears that we are one step closer to a resolution. It's early days but it appears that some of the power management settings for intel processors might be the issue. Intel Speedstep is a set of technology that is there to reduce the clock speed of your CPU in times when you don't need it so much. This is handy as it:
- Extends the battery time on a laptop
- Runs your CPU cooler, extending it's life
- Keeps your PC running more quietly because you don't need the fans running at such high speeds
- Reduces power consumption

Unfortunately, it seems this technology is giving gamers some issues and causing CPUs to run at 100%. Sounds familiar.
Disable Intel Speedstep technology - MajorGeeks Support Forums
THIS IS MADNESS! INTEL SpeedStep was holding my performance back! - Guru3D.com Forums
CPU Random Spikes 100% FIXED!!! - Forums - StarCraft II

These issues sound similar to the ones our customers have been getting. Like gamers, we want highest performance and we probably don't mind paying a few cents more a day in electricity if that means our screens don't freeze up.

Speedstep has to be switched off in the BIOS. My advice would be to leave this alone if you aren't having issues. If you are having issues, it might be because this speedstep is a bit glitchy and slowing your CPU down for you.

As well as Intel Speedstep - it also appears that "hyperthreading" in the bios settings can cause issues with running Ninja at peak times.

Many customers are experiencing significant performance improvements with hyperthreading switched off. I imagine this is because of the single-threaded nature of Ninja but I'm not sure.

If you have any questions about the products or services provided, please send me a Private Message or use the BMT "Ask Me Anything" thread
Reply With Quote
     
The following 10 users say Thank You to DionysusToast for this post:
     

Old August 26th, 2013, 11:35 AM   #3 (permalink)
Elite Member
Bangkok
 
Trading Experience: Intermediate
Platform: S5 Trader, NinjaTrader
Favorite Instrument: ES
 
DionysusToast's Avatar
 
Posts: 1,699 since Nov 2010
Thanks: 505 given, 3,867 received


My Documents Folder

By default, windows locates your My Documents folder on the system drive. This is normally no issue but both NinjaTrader AND GOMI files are located in your My Documents folder.

If your OS is on an SSD you might want to consider the following:
- your SSD has a finite life & that is governed by the amounts of times it writes to the HDD. Whilst GOMI & NinjaTrader aren't writing that much data, they can write to your disk constantly which will shorten the life of the SSD. I haven't done the math to figure out if this is significant or not.

If your OS os NOT on an SSD
- you really should have a second drive fitted and more the My Documents folder there. That way the data writes that Ninja performs do not compete with stuff your OS is doing. If it's all n a single drive, it can contribute towards disk thrashing.

For my own setup, I have the OS on an SSD and the My Documents on a secondary hard drive.

The method for moving the My Documents folder depends on the operating system. It's tricky on XP and pretty easy on Windows 7 & Windows 8.

Note that moving this folder WILL mean you have to change any references to DLLs within Ninja.

If you have any questions about the products or services provided, please send me a Private Message or use the BMT "Ask Me Anything" thread
Reply With Quote
     
The following 6 users say Thank You to DionysusToast for this post:
     

Old August 26th, 2013, 11:54 AM   #4 (permalink)
Elite Member
East Coast
 
Trading Experience: Beginner
Platform: Ninja
Favorite Instrument: ES, NQ
 
Posts: 125 since Apr 2010
Thanks: 55 given, 79 received

Windows 7 can fake the Documents location by using Junctions.

With XP, you can do the same but have to install MS' add-on

Junction

This can locate the directories to another drive while appearing to software to be in the original location.

Reply With Quote
     
The following 5 users say Thank You to Tarkus11 for this post:
     

Old October 9th, 2013, 06:14 AM   #5 (permalink)
Elite Member
Portland Oregon, United States
 
Trading Experience: Beginner
Platform: Ninjatrader®
Broker/Data: CQG, Kinetick
Favorite Instrument: Simulated Trading in my KLOWNBINE
 
Zondor's Avatar
 
Posts: 1,180 since Jul 2009
Thanks: 1,145 given, 2,107 received

Move files from HDD to RAM

There is an older thread about using a RAMDisk to improve Ninjatrader performance.
https://www.bigmiketrading.com/ninjatrader/23998-speeding-up-ninjatrader-ram-drive-ramdisk.html

The folders in the MyDocuments\Ninjatrader 7\db folder can be moved to a virtual disk implemented in RAM. When the computer shuts down, an image of the RAM disk is saved to HDD, and is automatically remounted upon restart. Junctions and symbolic links are used to fool Ninjatrader into thinking that the folders are in their default locations.

The Gom folder can also be in RAM, but it seems to make very minor demands on hard drive.

On my PC, this technique seems to be working reliably with the other changes that have been suggested in this thread.

"If we don't loosen up some money, this sucker is going down." -GW Bush, 2008
“Lack of proof that something is true does not prove that it is not true - when you want to believe.” -Humpty Dumpty, 2014
“The greatest shortcoming of the human race is our inability to understand the exponential function.”
Prof. Albert Bartlett

Last edited by Zondor; October 9th, 2013 at 06:22 AM.
Reply With Quote
     
The following 2 users say Thank You to Zondor for this post:
     

Old November 3rd, 2013, 04:22 PM   #6 (permalink)
Trading Apprentice
Cyprus
 
Trading Experience: Advanced
Platform: NinjaTrader
Favorite Instrument: Stocks
 
Posts: 4 since Feb 2011
Thanks: 2 given, 0 received

Disk thrashing question

Hi Dionysus

I read your post re improving performance in NT7 and have some questions. If you have some time to explain it will help me to understand how to move forward.

I am running a large number of strategies on US stocks, between 100 -200. NT7 can cope fine, there are no delays in execution (if there is any it is very small) but watching the Task manager RAM usage as the day goes by, the memory keeps building up.

I believe is because of the Trade info that is recorded in the SQL folder and the info that is written to the Logs.

I like your idea of moving the My Documents file but i don't quite understand how would this work. If i move the Ninjatrader folder to a different location, say the C:/, would'n't the program have problems launching?

Doesn't NT7 launch from that folder, using Templates, Workspaces and strategy defaults?

Also is there a way to stop Ninja writing all the trades to the SQL database and the Logs without moving the folder?

Thank you

Reply With Quote
     

Old December 30th, 2013, 02:55 AM   #7 (permalink)
Elite Member
Johannesburg - South Africa
 
Trading Experience: Intermediate
Platform: Sierra Chart
Favorite Instrument: ES
 
DarkPoolTrading's Avatar
 
Posts: 900 since May 2012
Thanks: 834 given, 1,039 received

Does anyone have any suggestions for reducing the amount of memory NT uses?

I load NT with only 2 charts of the same instrument both of which use GOMI tools and it uses almost 2gig memory! Im fully aware that NT is a resource hog but this is crazy. 2Gig memory to have 2 charts up.

I run Sierra Charts (my primary trading platform) with 6 charts up, ranging from volume profile, footprint, to regular candlestick charts. It uses about 700-800MB memory.

Any suggestions? Is there any way to narrow down precisely what is using so much memory in NT so I can disable it if im not using it?

Thanks.

You don’t trade the markets; you only trade your beliefs about the markets.
- Van K Tharp
Reply With Quote
     

Old December 30th, 2013, 03:32 AM   #8 (permalink)
Administrator: 'da Big Dawg
Manta, Ecuador
 
Trading Experience: Advanced
Platform: My own custom solution
Favorite Instrument: SP500, Stocks
 
Big Mike's Avatar
 
Posts: 35,620 since Jun 2009
Thanks: 23,526 given, 47,553 received


DarkPoolTrading View Post
Does anyone have any suggestions for reducing the amount of memory NT uses?

I load NT with only 2 charts of the same instrument both of which use GOMI tools and it uses almost 2gig memory! Im fully aware that NT is a resource hog but this is crazy. 2Gig memory to have 2 charts up.

I run Sierra Charts (my primary trading platform) with 6 charts up, ranging from volume profile, footprint, to regular candlestick charts. It uses about 700-800MB memory.

Any suggestions? Is there any way to narrow down precisely what is using so much memory in NT so I can disable it if im not using it?

Thanks.

How much historical lookback on each chart?

Mike

Due to time constraints, please do not PM me if your question can be resolved or answered on the forum.

Need help?
1) Stop changing things. No new indicators, charts, or methods. Be consistent with what is in front of you first.
2) Start a journal and post to it daily with the trades you made to show your strengths and weaknesses.
3) Set goals for yourself to reach daily. Make them about how you trade, not how much money you make.
4) Accept responsibility for your actions. Stop looking elsewhere to explain away poor performance.
5) Where to start as a trader? Watch this webinar and read this thread for hundreds of questions and answers.
6)
Help using the forum? Watch this video to learn general tips on using the site.

If you want
to support our community, become an Elite Member.

Reply With Quote
     
The following user says Thank You to Big Mike for this post:
     

Old December 30th, 2013, 03:48 AM   #9 (permalink)
Elite Member
Johannesburg - South Africa
 
Trading Experience: Intermediate
Platform: Sierra Chart
Favorite Instrument: ES
 
DarkPoolTrading's Avatar
 
Posts: 900 since May 2012
Thanks: 834 given, 1,039 received


Big Mike View Post
How much historical lookback on each chart?

Mike

Good point. A year on each. I just reduced that amount for the intraday chart and the memory dropped substantially. Im still not very impressed with the memory usage in comparison to my primary trading platform,...but at least NT is now usable. I'll just have to load less data

Thanks.

You don’t trade the markets; you only trade your beliefs about the markets.
- Van K Tharp
Reply With Quote
     

Old December 30th, 2013, 08:03 PM   #10 (permalink)
Elite Member
Houston, Texas
 
Trading Experience: Beginner
Platform: NinjaTrader
Favorite Instrument: futures
 
learning0101's Avatar
 
Posts: 178 since Aug 2011
Thanks: 2,795 given, 142 received

elaborate please



DionysusToast View Post
My Documents Folder

By default, windows locates your My Documents folder on the system drive. This is normally no issue but both NinjaTrader AND GOMI files are located in your My Documents folder.

If your OS is on an SSD you might want to consider the following:
- your SSD has a finite life & that is governed by the amounts of times it writes to the HDD. Whilst GOMI & NinjaTrader aren't writing that much data, they can write to your disk constantly which will shorten the life of the SSD. I haven't done the math to figure out if this is significant or not.

If your OS os NOT on an SSD
- you really should have a second drive fitted and more the My Documents folder there. That way the data writes that Ninja performs do not compete with stuff your OS is doing. If it's all n a single drive, it can contribute towards disk thrashing.

For my own setup, I have the OS on an SSD and the My Documents on a secondary hard drive.

The method for moving the My Documents folder depends on the operating system. It's tricky on XP and pretty easy on Windows 7 & Windows 8.

Note that moving this folder WILL mean you have to change any references to DLLs within Ninja.

Hi @DionysusToast
Happy Holidays
Thanks for the info here that you have provided. Am about to change location of "mydocuments" folder to another drive. If you could please elaborate or point me in a direction to see how-to, about the "Note that moving this folder WILL mean you have to change any references to DLLs within Ninja" How does one do that?
Thanks in advance


Edit: Maybe I understand now after a little looking--
Is what you are referring to inside of NT when you edit a indicator and add references??


Last edited by learning0101; December 31st, 2013 at 05:56 PM. Reason: may have figured it out
Reply With Quote
     

Reply



Tags
00, advice, antivirus, art, bat, bot, chart, control, data, database, dom, don, dow, dtn_iqfeed, el, enter, es, file, gomi, guru, historical, indicators, iqfeed, lookback, lot, management, market, market replay, market_replay, method, microsoft, mind, monitor, ninja, ninjatrader, ninjatrader 7, operating system, options, order, performance, physical, replay, save, scan, settings, setup, software, sql, start, strategies, strategy, suggestions, support, system, tick, time, trades, virtual, windows
     

Big Mike's Trading Forum > Trading Forums > Platforms and Indicators > NinjaTrader > Ninja Tunes

Thread Tools Search this Thread
Search this Thread:

Advanced Search


Similar Threads
Thread Thread Starter Forum Replies Last Post
Ninja Indicator to Ninja Strategy emini_Holy_Grail NinjaTrader Programming 10 March 10th, 2013 12:23 AM
value in Kinetick for ninja mcteague Brokers and Data Feeds 2 December 17th, 2012 07:06 PM
Switching Fin.Alg TPO Market profile from Ninja 6.5 to ninja 7 MadManmos Vendors and Product Reviews 19 August 23rd, 2011 09:15 AM
Help with Channel This in Ninja 6.5 CaptainAmericaXX NinjaTrader 20 March 16th, 2011 05:38 PM
Ninja 7 Mauro_60 NinjaTrader 17 July 13th, 2010 09:18 AM


All times are GMT -4. The time now is 11:34 PM.

Powered by vBulletin®
Copyright ©2000 - 2015, Jelsoft Enterprises Ltd.
SEO by vBSEO ©2011, Crawlability, Inc.
Copyright © 2015 by Big Mike Trading. All information is for educational use only and is not investment advice.
There is a substantial risk of loss in trading commodity futures, stocks, options and foreign exchange products. Past performance is not indicative of future results.
 
no new posts

Page generated 2015-01-02 in 0.13 seconds with 20 queries on phoenix via your IP 193.202.110.188