What does
Pip Cop do?

Icon/Color key
Ordered by equity
RoboMiner LIVE
$2229 · $-5,454 · $5/d
    Updated : 2010-03-13 10:00am CST
Detailed Stats


Go Back   Home > Forums > General forums > Trading Programs

» Welcome to the Pip Cop - Forex Robot Reviews and forums.
PipCop reviews MetaTrader Forex robots (EA's) in real-time and posts detailed statements every 15 minutes.
We ONLY forward test on real accounts for the most accurate robot reviews!

Be smart! Read the Review FAQ or you WILL lose money!
If you enjoy the site, please let me know by registering or donating! Thanks! -- PipCop
P.S. This message is hidden if you register!

Trading Programs Trading software and tools to help you manage robots, run reports, etc.
Reply
 
LinkBack Thread Tools Display Modes
Old 2009-04-15, 07:48 PM   #1 (permalink)
Site Supporter
 
rsmereka's Avatar
 
Trader for 4 - 8 years
Location: Southwestern Ontario, Canada
Posts: 519
My Trading Journal


Trading Live with:
MegaDroid, EB-Turbo and The Channel Scalper
Default MT4Analysis Project

I would like to start off this forum by announcing the MT4Analysis Project. This project will (eventually) deliver end-to-end analysis of MetaTrader V4 trading systems.

The backend consists of a program to parse MetaTrader V4 statement files into a database. The frontend will consist of a series of programs to analyze and present various trading performance statistics.

I have chosen to write this project using the Php language, SQLite as the backend database and Php/HTML/Javascript as the frontend.

The first part of this project has been written and is in alpha stage (working but not perfected). The first part is a Php program to parse an MT4 statement file and load an SQLite database from its contents.

The attached 'zip' file contains eveything you need to load the database. The parser program called 'mt4_sqlite_loader' is a Php CLI program. It is run from the command line or scheduler. It is not an HTTP program run from a web server.

Requirements:
Since Php is a server-side scripting language, you will need a web server with Php and Php CLI installed on your Windows PC (this is not as hard as you think...read on).

-Web server with Php module
-Php V5
-Php CLI capabilities
-Php PDO extension
-Php PDO SQLite V3 extension

This may sound complex but it does not have to be. The really good news is that there is a Windows distribution of LAMP called WAMP that just works right out of the box.

If you decide to use WAMP, download the older 1.7.4 version. I have had problems with Php CLI and the SQLite database driver with newer versions. If you get a newer version to work with Php CLI, please let me know.

Assumptions:
I have used the Php PDO interface which should greatly ease the ability to moving the backend to another database like MySQL.

The dates and times are loaded into the database as is (except for removing the periods). This will allow us to view the trades in a different time zone by calculating the new date and time on the fly.

Quick Start:
1. Make sure you have the requirements installed as above.
2. Create a new directory at the root level of the 'C' drive called 'mt4_sqlite_loader' and unzip the attached zip into it.
3. Copy the 'skelton.db' file located in the 'db' directory to another file like 'live.db' or 'demo.db'. The 'skelton.db' is a skelton of the SQLite database file with the tables defined without any trade data.
4. Crank up the loader:
a. start up a command prompt
b. change to MT4 loader directory by typing:
cd \mt4_sqlite_loader
c. parse and load with the command:
\wamp\php\php mt4_sqlite_loader.php db\live.db detailedstatement.htm

The above command assumes:
1. You are using WAMP installed in the default location.
2. The database file you decided to use is called 'live.db' in the 'db' sub-directory.
3. Your MT4 statement file has been placed into the 'mt4_sqlite_loader' directory.

The layout (schema) of the database file is in the file 'schema.txt' in the 'db' directory.

Only the loader exists right now but if you know SQL, you can download the SQLite CLI program and type any queries that you want. Here are some I have been using:

-To verify the closing balance, first get the record with the highest ticket number:
select ticket from closedtrade where max(ticket);

Then, once you know the ticket number, display the record:
select * from closedtrade where ticket = ticketnum;

Where 'ticketnum' is the ticket number from the first query.

-To find all the different EA's in use in this account:
select distinct magic from closedtrade;

-To see how each EA is performing (total profit)
select total(pl) from closedtrade where magic = 123456;

Where '123456' is the EA magic/reference number that you got from the previous query.

One of the bad parts to MetaTrader is that the broker can decide to consolidate your trading records and then the statement file is cleared of all previous trades. By using this database method, you will never loose any data.

The loader will only load closed trades that do not exist in the database. The ticket number is the key of the 'closedtrade' table and it must be unique. You can run the loader program as many times as you want.

I need some help with the loader program. I have tested it with some statement files but, obviously, not not covered every possibility. If you encounter any bugs using the loader please let me know.

Stay tuned for more.
Enjoy,
Rick
Attached Files
File Type: zip 20090415.zip (43.7 KB, 7 views)
rsmereka is offline   Reply With Quote
   
 
Old 2009-04-16, 09:21 AM   #2 (permalink)
Site Supporter
 
rsmereka's Avatar
 
Trader for 4 - 8 years
Location: Southwestern Ontario, Canada
Posts: 519
My Trading Journal


Trading Live with:
MegaDroid, EB-Turbo and The Channel Scalper
Default Two Bugs

There are at least two bugs in the mt4_sqlite_loader program.

The first bug should appear if you attempt to load an MT4 statement file with open or pending trades. I think the program will attempt to load the open or pending trades into the database. I forgot to look for the html table end delimiter.

The second bug will manifest if your broker consolidates your trades and your statement file is thus cleared. When you attempt to load an MT4 statement file with some closed trades (none of which are already in the database) without any balance transaction, mt4_sqlite_loader will not be able to set the opening balance properly and the closed trades will be written with the wrong current balance.

I will be working on these two bugs and will release an update as soon as preliminary testing is complete.

Rick
rsmereka is offline   Reply With Quote
Old 2009-04-16, 03:02 PM   #3 (permalink)
Site Supporter
 
rsmereka's Avatar
 
Trader for 4 - 8 years
Location: Southwestern Ontario, Canada
Posts: 519
My Trading Journal


Trading Live with:
MegaDroid, EB-Turbo and The Channel Scalper
Default

Turns out that there were plenty of bugs in that initial alpha release of mt4_sqlite_loader. I squashed them all. Changes:

1. The program now knows when to stop scanning for trades although I need to make sure that this assumption remains stable.
2. I removed the 'balance' field from the 'closedtrade' table. Calculating and writing the balance for every trade was causing problems especially when (like Pipcop does in the Robominer live account statement on mt4stats.com), the closed trades in the statement file are in descending order (most recent first). We will be able to get the balance at any point by calculating on-the-fly anyway.
3. I was trying various statement files and the ones that are from IBFX using a mini account were causing a problem. IBFX in it's mini account puts an 'm' at the end of the currency pair . I removed the 'm' at the end before loading into the database.
4. Some of the test statement files I was trying came from mt4stats.com which started to cause another problem. You see mt4stats.com modifies the statement file. Any html tag with parameters (like <td title="blah">) in the original MT4 statement file does not quote the parameters (like <td title=blah>). This is technically wrong but I programmed the parser to accept this. When I tried parsing a statement file from mt4stats.com, I realized that they 'fix up' the statement file by quoting all parameters in tags. As a result, I had to make my parser more flexable by accepting both methods.

Bizzarroo but cool tip of the day:
I am using Php 'fopen' function to open the statement file. This function will accept a URL, like this:
php mt4_sqlite_loader db\elbizarroo.db http://www.website.com/statement.htm

Using this method will get mt4_sqlite_loader to parse the statement file directly from a web site. This method, however, mostly does not work because most webmasters including mt4stats.com do not allow programs to start opening files on the web server. This is generally considered a security risk.

On the other hand, try this method:
php mt4_sqlite_loader db\toocool.db ftp://user:password@example.com/somefile.txt

Here is the command I just tried and it works like a charm:

php mt4_sqlite_loader db/live.db "ftp://rsmereka:pass@ftp.mt4stats.com.../statement.htm"

The double slashes before the file name is required. Note that Pipcop's interface is mangling my examples.

Hey Pipcop. This #&*@&38 interface keeps mucking up my examples. How do I turn this off for a couple of words or sentences. I turned off 'automatically parse links' but it keeps on mangling by examples treating them as links. I tried quoting the example also. No joy.

Rick


Attached Files
File Type: zip 20090416.zip (13.4 KB, 2 views)
rsmereka is offline   Reply With Quote
Old 2009-04-16, 03:27 PM   #4 (permalink)
Site Supporter
 
rsmereka's Avatar
 
Trader for 4 - 8 years
Location: Southwestern Ontario, Canada
Posts: 519
My Trading Journal


Trading Live with:
MegaDroid, EB-Turbo and The Channel Scalper
Default HTTP Direct Loading Works

I got mt4_sqlite_loader to work directly from a statement file posted on mt4stats.com. Here's how:

php mt4_sqlite_loader db/live.db http://rsmereka.mt4stats.com/

The trailing slash is required.

This will allow you to load up a trade database based on any published statement file

Rick
rsmereka is offline   Reply With Quote
Old 2009-04-20, 02:24 PM   #5 (permalink)
Site Supporter
 
rsmereka's Avatar
 
Trader for 4 - 8 years
Location: Southwestern Ontario, Canada
Posts: 519
My Trading Journal


Trading Live with:
MegaDroid, EB-Turbo and The Channel Scalper
Default

Here is the next version of mt4_sqlite_loader. This version has seen a lot of changes mostly to the database. Firstly, an account database has been added. It will contain a list of every Metatrader account in use. Each account will have it's own trade database. The following tables are part of the trade database:

broker
currencyPairSet
currencyPair
transactionType
ea
closedTrade

The broker table contains a list of all brokers in use including the time zone that the broker uses and the currency pair set in use. A currency pair set is a series of currency pair identifiers for a specific broker or multiple brokers. I have already loaded two currency pair sets (standard lot and IBFX mini) and two brokers (Alpari UK and IBFX). The ea table should also be loaded with the names and magic numbers of all EA's in use.

The MT4Analysis project package file is quickly approaching the maximum allowable by a 'zip' file on this site. I will next be moving the central location of the MT4Analysis project package file to my website. I will also be constructing some badly needed documentation along with building the database management screens.

Rick
Attached Files
File Type: zip 20090420.zip (39.2 KB, 14 views)
rsmereka is offline   Reply With Quote
Old 2009-04-20, 02:42 PM   #6 (permalink)
Pip Chief of Police
and Site Owner
 
PipCop's Avatar
 
Trader for 2 - 4 years
Location: Sterling, VA, USA
Posts: 1,239
My Trading Journal


Trading Live with:
RoboMiner Pro (2 instances)
Default

Quote:
Originally Posted by rsmereka View Post
The MT4Analysis project package file is quickly approaching the maximum allowable by a 'zip' file on this site. I will next be moving the central location of the MT4Analysis project package file to my website. I will also be constructing some badly needed documentation along with building the database management screens.
I believe the maximum zip upload is 5mb which should be plenty of space. However, I am always a fan of one file in one place, rather than updated files scattered throughout a discussion thread!
__________________
> Don't get busted by PipCop ... Read the Rules!
> Questions? Please read the REVIEW FAQ first!

> Have a Forex link? Add it to the Links Directory
> How to link your MT4 statement to your profile
PipCop is online now   Reply With Quote
Old 2009-04-20, 03:47 PM   #7 (permalink)
Site Supporter
 
rsmereka's Avatar
 
Trader for 4 - 8 years
Location: Southwestern Ontario, Canada
Posts: 519
My Trading Journal


Trading Live with:
MegaDroid, EB-Turbo and The Channel Scalper
Default

Quote:
Originally Posted by PipCop View Post
I believe the maximum zip upload is 5mb which should be plenty of space. However, I am always a fan of one file in one place, rather than updated files scattered throughout a discussion thread!
I think your right. Five megabytes should be ample especially when all most of the files are just text.

To tell you the truth, I was/am mostly concerned with the usual problem of a long thread based on software. There are files scattered all over the thread. This happened to me just the other day researching an EA. There were so many versions and variations, I eventually gave up.

Rick
rsmereka is offline   Reply With Quote
Old 2009-07-25, 06:43 PM   #8 (permalink)
Site Supporter
 
rsmereka's Avatar
 
Trader for 4 - 8 years
Location: Southwestern Ontario, Canada
Posts: 519
My Trading Journal


Trading Live with:
MegaDroid, EB-Turbo and The Channel Scalper
Default

Refer to this thread regarding the new mt4stats.com statement format which has a direct impact on this project.

Rick
rsmereka is offline   Reply With Quote
Old 2009-10-25, 03:00 PM   #9 (permalink)
Site Supporter
 
rsmereka's Avatar
 
Trader for 4 - 8 years
Location: Southwestern Ontario, Canada
Posts: 519
My Trading Journal


Trading Live with:
MegaDroid, EB-Turbo and The Channel Scalper
Default New Mt4Analysis Package Available

After months of work, I have just posted a new version of the MT4Analysis project. You can always download the current version of MT4Analysis here.

This new version has the complete database structure finalized, all the configuration and management screens complete, some of the documentation in place and a full SQL command line available to run queries.

I have also put online, a live, read-only copy of MT4Analysis here. You can view the entire sample configuration and run any SQL queries that you like against a total of six trading databases (5 demo and one live). I plan to update the databases on a daily basis.

Enjoy,
Rick

Last edited by rsmereka; 2009-10-25 at 03:10 PM. Reason: Spelling
rsmereka is offline   Reply With Quote
Old 2009-10-29, 12:45 PM   #10 (permalink)
Site Supporter
 
rsmereka's Avatar
 
Trader for 4 - 8 years
Location: Southwestern Ontario, Canada
Posts: 519
My Trading Journal


Trading Live with:
MegaDroid, EB-Turbo and The Channel Scalper
Default

Some updates for you:

1. I just posted a documentation change to the live beta system here. The documentation is sloooowly getting there. I hate documenting

2. I found a bug in the live beta system which was caused by my web hoster having Php magic quotes turned on. Since magic quotes cannot be turned off at runtime in the code and the .htaccess directive to turn magic quotes off was not recognized on my web hoster's server, I had to modify the MT4Analysis SQL command line page to remove the extra quotes after the page has been submitted.

3. Since I do most of my coding using Ubuntu Linux, I created some shell scripts to:

a. read the MT4 statements and load them into the database
b. use ftp to post the database to my live beta system

I then created a cron job to automatically update all the databases every hour on the hour during trading. The result is that the databases in the live beta system should be accurate to within an hour.

I have not yet built new download packages containing the above changes.

Current version is .4.03-2009.10.29

Rick
rsmereka is offline   Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes





All times are GMT -5. The time now is 11:06 AM.

Powered by vBulletin ®
Site content and design Copyright © 2009 PipCop, LLC.
Site designed, hosted, and maintained by One Web Ave
Template by vBSkinworks.
Content Relevant URLs by vBSEO 3.3.2