Getting Market Data¶
Market Data retrieval is done by a piece of software called a Blotter. A Blotter connects to the broker (in QTPyLib’s case, Interactive Brokers via TWS/IB), handles incoming market data and passes it to the algo for processing.
Blotters optionally (but usually) also take care of storing the market data in a Database for later analysis, back-testing new strategies, etc.
In QTPyLib’s case, the Blotter handle all of the above, while your algorithms subscribe to the Blotter’s updates via pub/sub mechanism using ZeroMQ - a blazing fast Message Queue.
QTPyLib was designed with a “One Blotter To Rule Them All” design in mind. All your algorithms can listen to a single Blotter running in the background without a problem and without consuming any unnecessary system resources. Simply put: do not run multiple Blotters unless you have a very specific reason to do so.
Creating the Database¶
The first thing you need to do is to create the MySQL database where your Blotter will store tick and minute data for later use.
Once you’ve created the database, note its name for the next step. The Blotter will automatically create the required database tables when it runs for the first time.
Writing your Blotter¶
To get started writing your Blotter, you’ll need to create a Blotter object sub-class and name it.
Then, inililize your Blotter by passing your MySQL credentials and TWS/IBGW port and run it.
# blotter.py from qtpylib.blotter import Blotter class MainBlotter(Blotter): pass # we just need the name if __name__ == "__main__": blotter = MainBlotter( dbhost = "localhost", # MySQL server dbname = "qtpy", # MySQL database dbuser = "master", # MySQL username dbpass = "blaster", # MySQL password ibport = 4001, # IB port (7496/7497 = TWS, 4001 = IBGateway) orderbook = True # fetch and stream order book data ) blotter.run()
Running your Blotter¶
With IB TWS/GW running, run the Blotter from the command line:
$ python blotter.py
Initilizing via CLI¶
You can also override the initilized paramters (or ommit this part of the code altogether) and pass runtime parameters using the command line.
In this case, your code would look something like this:
# blotter.py from qtpylib.blotter import Blotter class MainBlotter(Blotter): pass # we just need the name if __name__ == "__main__": blotter = MainBlotter() blotter.run()
Then, run the Blotter by passing the parameters via the command line:
$ python blotter.py [--dbport] [--dbname] [--dbuser] [--dbpass] [--ibport] [--orderbook] [...]
Below are all the parameters that can either be set via the
or via CLI:
--symbolsCSV database of IB contracts for market data fetching (default:
--ibportTWS/IBGW Port to use (default:
--ibclientTWS/IBGW Client ID (default:
--ibserverIB TWS/GW Server hostname (default:
--zmqportZeroMQ Port to use (default:
--zmqtopicZeroMQ string to use (default:
--dbhostMySQL server hostname (default:
--dbportMySQL server port (default:
--dbnameMySQL server database (default:
--dbuserMySQL server username (default:
--dbpassMySQL server password (default:
--dbskip[flag] Skip MySQL logging of market data (default:
--orderbook[flag] Tells the blotter to fetch and stream order book data (default:
--dbpassMySQL server password (default:
--max_threadsMaximum number of threads to use (default is 1)
It’s recommended that you set the
max_threads parameter based on your strategy’s needs and your machine’s capabilities!
As a general rule of thumb, unless you’re subscribing to 100+ instruments, you probably don’t need to tweak this parameter.
Once your Blotter runs for the first time, you’ll notice that a new
symbols.csv has been created in the same directory
as your Blotter.
This fill will store all the instruments that algos connecting to this Blotter will request data for. Your blotter will keep logging market data for these instruments even when you stop your algos so you have continious market data stored in your database for future research and backtesting (expired product will be deleted automatically from this file).
You can, of course, add or delete unwanted instruments from the CSV file manually at any time – without stopping your Blotter.
Eample a populated
symbol,sec_type,exchange,currency,expiry,strike,opt_type AAPL,STK,SMART,USD,,0.0, ES,FUT,GLOBEX,USD,201609,0.0, NFFX,OPT,SMART,USD,20160819,98.50,PUT
This file’s structure is better understood when looked at as a table:
With your Blotter running, its time to write your first Algo…