Tuesday, July 2, 2013

Modularization by extending CAction and using Widgets

Today I will explain how we can modularize certain repeated functionalities using CAction and Widgets. For this case I use one functionality in our project named as ReportAbuse.
Well those are not the only things you will learn in this post. There are multiple other concepts like BaseModel, ExternalActions, Enums etc.,

Business Problem : In earlier stages we used to write report abuse related html and also its companion JavaScript for its behavior. At that time the HTML was of 20-30 odd lines and JavaScript was also about the same so we never had much problem working with it. This process was fine for couple of months but after that we are involved in a situation where we got extra business object (module) for every month so our beloved report abuse code blocks are pretty heavy to maintain in each and every location. Over the time developers are trying to add bells and whistles to that code block to their liking if they have to implement in different place (business object or module) which created fragmentation within the simple code block of report abuse. Spin forward this situation for 3 months and we have 5-10 different templates just for report abuse. This is a big time problem for new developers and existing ones too.

Solution : Since we have started with a single code block for report abuse and eventually ended up with more templates I wanted to standardize the code by using only one template for all. So I wanted to provide this behavior by using widgets combining with other features provided by Yii which I mentioned above.

I have posted some code for you people which consists of modified version of my working code for your ease. In modified version I might change things to generalize to Yii or end up writing some TODOs(Implement these TODOs to your liking and your project requirements).

I have posted the code on BitBucket and one can download it from here
Update : Sorry people above link doesn't give you exact information. Please look into this

Brief explanation about each file, why it is used and what else you can improve on it.
  • ReportAbuseForm.php
This is base form for all report abuse forms, I mean I will use only this form for every request of report abuse throughout application.
Here $context is important and very helpful when dealing with custom business requirements. We can send any information in that $context variable to do some other process and that is the reason it has been an array because we might need to send some heavy customized parameters in future. One can also define different labels for the same form in different locations and the $context variable comes in handy. In questions you may need ‘Comment’ and in groups you may need ‘description’ we can do this by using objectType or context.
  • ReportAbuse.php (Widget)
This is the main code which handles all display of HTML and its behavior using JavaScript. One need to observe that in code I have used $context for profile to display the widget in small size. You can do more customization if you require.
Not all times we need parentId or context so we bind them only if things are set and they are passed separately to customize UI blocks and report abuse thing is sent using $reportAbuseForm for which all necessary data is binded.
  • reportabuse.php (Widget view)
It uses $params to customize UI blocks to render. FormId is made unique so that it doesn’t conflict in DOM. The first part of the script handles all the behavior of sending data through Ajax, validating it, binding errors etc., The second part handles all the behavior of animations when buttons are clicked, clearing forms etc.,. We have used Yii Messages to do string externalization but you can write it there itself but I strongly recommend using string externalization mechanism provided by Yii.
Based on widgetSize different CSS classes have been written to get different effect at different places. Most of the data is set already using hidden Fields and we used this view file only to get user comment about why he is reporting this object.
  • Report.php (External Action by extending CAction)
This is the action called every time we report any object. So changes in this action will reflect throughout the application. To use this action we need a controller so we got it by using $this->getController(); We then validate the form for our rules in ReportAbuseForm.php for null, length validations (If required put some other rules too). Then we used $siteModel to access database and insert the report abuse. We have used this model to consolidate all common logic's like this to one place. We then generalized this method to serve report abuse for all objects (like questions, answers, groups etc.,). Since we are validating only on comment attribute we get errors only for that so we set that error in the error case.
Note : I haven’t tried this scenario with multiple attributes which take user input.
  • SiteModel.php
This is the model which handles all common logics like report abuse, tag search, comment insert (Will discuss this in later post).
reportAbuse method takes $reportAbuseForm and $context as parameters and generates input parameters required for stored procedure (we use stored procedures and java as data layer and yours may be different) then we call the java data methods with necessary parameters to get the job done. In your cases you might need to generate parameters for each stored procedure and call them in switch case or else create a single SQL statement which can handle all those cases or there might be a stored procedure which handles all those logic by giving necessary inputs to it. It all depends on your data access methodologies.
  • SiteController.php
The thing which glues what all we have discussed above is defining this external action to use in our application and we can do this by defining our external action in actions array. We need to initialize our external action like this in actions array
'report'=>'application.components.actions.Report',

By defining like this we get report URL like this <domain name>/report. It might help even in SEO stuff.
Example: http://192.168.0.31/report
More about TagsProvider and CommentsProvider in later posts.

One can use this component by initializing it from view file like this (we needed tipsy but you might not need it)

<div class="reportAbuseWrapper">
    <?php
    $this->widget('ReportAbuse', array(
            'params'=>array(
            'objectId'=>$qaViewModel->question->id,
            'objectType'=>'question',
            'tipsyClass'=>'tipsyleft',
            'authorId'=>$qaViewModel->question->author->id,
           'reportStatus'=>$qaViewModel->question->reportAbuse
        )
    ));
    ?>
</div>


Here is the widget in action

 
This is how the widget looks with all its CSS and JavaScript intact.


This is widget after clicking Report Abuse link in blue color


This is the widget after form validations. Here it is null validation. I mean clicking submit without entering any description.


This is widget after form validations and this time it is maximum allowable characters validation done.


This is confirmation that it is reported and the application will take care of the rest.


This is the URL which it will hit. Of course mine is different. This is the data we are sending for report abuse action.


This is the response after the controller action is called .



One can look into all files which I have mentioned above and change to your requirements.
Please do post your comments on this. If you have any doubt please contact me for more details.

Tuesday, June 25, 2013

Setup apigen to generate documentation

Lately I'm working on some extensions to improve my productivity and also add value to my project.
In the process I tried to implement unit test cases for my project and test them using PHPUnit.
One can find information on that on my post here

Now I'm trying to get some automation to make documentation for my project.
Well in general cases it will be mundane task for each successful build.
People who have seen my last post might think that I'm going to blog about phpDocumentor2 but things are different this time.
I have tried more than couple of tools to generate documentation for my project like
  1. yiidocsgenerator
  2. phpDocumentor
  3. phpDocumentor2 (works for basic implementations)
  4. apigen (working for me)
You might want to know why I didn't used above tools to get my documentation done
Before that I need to tell you that our project was a mess in terms of standards so no documenting tool is ready to give perfect output. Our mess includes not managing namespaces correctly for user, admin, vendors end.
  1. yiidocsgenerator - Gives errors related to multiple declarations of same class in different places. Gives problem if there is a class which extends a base class and it couldn't get it at parsing time.
  2. phpDocumentor - Old UI. More info is thrown out at random places. Not intuitive to read documentation. It gives a feeling that we are reading code and not documentation.
  3. phpDocumentor2 - Good UI. For basic documenting it works but when we use configuration file then we might land into problems regarding parser, transformer, target directory.
  4. apigen - Works flawlessly and has decent UI to get going.
Leaving apart my failures with other documenting tools I will post my success story with apigen here

One can always look at official documentation

Steps to get documentation done
  1. Setup/Installation
  2. Basic testing of command
  3. Generalizing using config file
  4. Testing command using config file

Setup/Installation


This command tells that it needs to resolve any dependencies it faces while installing below package
So we don't have the work of setting up all dependeny packages and their channels.
pear config-set auto_discover 1

Install apigen package for documentation
pear install pear.apigen.org/apigen

Basic testing of command


This is a simple execution to prove that this can document our project
apigen.bat --source C:\Users\sampathv.IE\Documents\Projects\<your project>\protected\components --destination C:\Docs\<your project>\apigen\Doc
Check the destination for index.html. See if the documentation is fine. 

Note: Here I'm documenting only components of my project because it is test and it has less files to parse.

Issue: When I executed only for components, documentation was successful but when I tried whole protected folder then there is an issue related to insufficient memory.
The error was asking me to increase my memory allocated to PHP for processing large amount of files.
Solution: So while researching I came to know that I need to change memory related values in php.ini config file. Initially it was 128M for my installation for which I changed to 512M.
For more information regarding how and where to edit please see these links
http://www.hostucan.com/webmaster-tutorials/increase-php-memory-limit
https://groups.google.com/forum/#!forum/apigen

 

 Generalizing process with config file


We can setup our config file using this official documentation and sample config file

When I used information from both links I faced certain problems like

Issue : I faced certain problems regarding skipDocPath parameter too long (more than 260 characters) because I added more than 15 directories to omit while generating documentation.
Well I could able to solve the issue by reducing the overall directories but there is another problem.
The documentor doesn't understand the skipDocPath parameter and it does document all files in specified project path which is not what you want at times.
Solution : Well I'm still looking into this issue. I think the problem is with relative paths and I don't understand how to give those things in windows environment. Everywhere the documentation is given for only linux paths and no documentation for windows.

After a lot of trials and tinkering with NEON Parser I could able to build some config file for my own project and you can find it here

Note : This is the first time I'm dealing with Git so if any users not able to view the file please comment and I will try to provide it by other means. Please change necessary locations in config.neon to your environment.

Testing command using config file

apigen.bat --config .\config.neon
After the command completes open up index.html page and check if that is what you intended.

Note : I executed the command from the directory where config.neon exists. Make sure you too do that.

Sunday, June 23, 2013

Useful PEAR commands to get things done

Today I'm going to outline some important commands related to PEAR.
These commands are nothing new which you can't find online or in PEAR site but these are something which I need every time to get things done on my workstation for any new extension.

For more information about PEAR follow this link
Here are some important commands

Check which channels are setup on your PEAR installation
pear list-channels

Consider a channel to look for packages in current session.(add or use this channel)
pear channel-discover pear.phpdoc.org

Check available packages in a channel.
pear remote-list -c pear.phpdoc.org

Install phpDocumentor2 package
pear install phpdoc/phpDocumentor-alpha

List all installed packages in a channel
pear list -c pear.phpdoc.org

List all available packages in a channel
pear list-all

Get information of files which are installed through a package
pear list-files phpDocumentor-2.0.0b4

Clear cache (I often use it when I get errors during installations)
pear clear-cache

Uninstall a package (mention complete package name otherwise it throws errors regarding package not available)
pear uninstall phpDocumentor-2.0.0b4

Friday, June 14, 2013

Setup and Write Yii Test Cases

First you need to know below things to start running a test case.

Business Problem:
Generally developers develop a web application or standalone application then they give the build to testers to test it. So testers understand all features of the build and plan to test it but when the build is so big then testers feel uncomfortable to test large builds. These scenarios come more than once before the build is live. So eventually a developer or tester come to a point of automating the whole thing which brings us to the point of writing a test suite.
One can start a test suite by using third party components or write individual test cases specifically for their business.

Here we are interested only in specific test cases which are related to business so we talk about only those in further discussions.

Solution:
Yii framework provides a base platform where in developers can test their features by writing separate code which tests the real code.
While creating a new project yiic tool will also generate necessary folder structure where in we can put test related files. (I guess you people used that tool)
For more information Yii Testing

Steps involved
  1. Installation/prerequisites
  2. Setup project for testing
  3. Write Test Cases
  4. Run Test Cases

Setting up the environment:

My environment is something different when compared to others.
  • We have code in windows and work on code development in eclipse
  • We share the folder and then map the folder to a Linux share.
  • We then symlink the code folder so sites
  • Use Apache as web server to serve the requests.
On whole we develop code in windows using eclipse and use Linux to host the code. We use share folders to sync the changes from windows to Linux.

Installation


 For testing we don't require that complex setup


Get PHP runtime
Goto below url
http://php.net/downloads.php
Download 5.3.x version for compatibility reasons (I have that version)
My file looks something like this 'php-5.3.26-Win32-VC9-x86.zip'
Extract the archive.

Get PEAR package manager
http://pear.php.net/go-pear.phar
Download the phar file.
Copy the go-pear.phar file to 'C:\Users\<user>\Downloads\php-5.3.26-Win32-VC9-x86' directory
Now open command prompt (I'm using power shell for all operations) and execute go-pear.phar using php runtime
Note: From this point I consider shell as your command shell whether it is command prompt or power shell
.\php.exe .\go-pear.phar

 If it asks for 'local' or 'system' then go for system

To select all options press enter

To install registry keys execute
.\PEAR_ENV.reg

Install Selenium RC Server as documented in Yii
http://selenium.googlecode.com/files/selenium-server-standalone-2.33.0.jar

Well if version changes we can always look at here http://docs.seleniumhq.org/download/ to get latest ones

Note: A working java environment should be there for Selenium Server to run because its a jar file.



Now install PHPUnit
.\pear.bat channel-discover pear.phpunit.de
.\pear.bat install PHPUnit

Install SeleniumTest
.\pear.bat install phpunit/PHPUnit_Selenium

If you run into any errors like "No releases available for package "pear.phpunit.de/PHPUnit_Selenium"" then run below command
.\pear.bat clear-cache


Setup project for testing

Goto directory <your web application folder>/protected/tests
Open WebTestCase.php and change the TEST_BASE_URL in eclipse or text editor
define('TEST_BASE_URL','http://192.168.0.32/<my web application>/index-test.php/');

Note: I have already mentioned that my environment is different. In your cases it my be localhost or 127.0.0.1

After that open the url and see if there are any other errors with main config with index-text.php
(I have to do some other config because we have more things in index.php which aren't copied to index-test.php)


Write Test Cases

Yii made this easy by providing some basic test cases where we can understand them and implement the logic to our scenarios.

For sample test cases open below file

protected/tests/functional/SiteTest.php

One can start editing those test cases or write their own.
I have written my own for my specific requirement. Here is the code snippet for your reference
public function testContact()
{
    $this->open('?r=site/contact');
    $this->assertTextPresent('Contact Us');
    $this->assertElementPresent('name=ContactForm[firstName]');
    $this->type('name=ContactForm[firstName]','tester');
    $this->assertElementPresent('name=ContactForm[lastName]');
    $this->type('name=ContactForm[lastName]','beta');
    $this->type('name=ContactForm[email]','demo.123@gmail.com');
    $this->type('name=ContactForm[purpose]','other');
    $this->type('name=ContactForm[body]','PHPUnit test cases sample');
    $this->click("//input[@value='Submit']");
    $this->waitForTextPresent('Thank you for contacting us.');
}
To change test case preferences open phpunit.xml file and change as per your requirements.
If you feel you don't want to test in Internet Explorer then you can remove Internet Explorer browser node or you can add another browser to your configuration


Running Test Cases

Now start running test cases against PHPUnit using Selenium Remote Control Server

Before that make sure that we are running an instance of Selenium RC Server
For this case open another shell and type this command to run it
java -jar .\selenium-server-standalone-2.33.0.jar

 Leave that window because it will throw log information which may be useful.

Open another shell window for running test cases

Change directory to your project folder
phpunit.bat .\functional\SiteTest.php

Note: Add phpunit.bat location to PATH in environment variables to access it anywhere. It it suggested to add

Note: This is how I made PHPUnit to work on my workstation and run test cases of my project. Your experiences may vary. This post is purely meant for informational purposes and I don't guarantee that everything mentioned here works as expected.

For more information browse below links
Yii Testing
PHPUnit Documentation
PHPUnit Manual

Thursday, June 13, 2013

Recover eclipse code templates using workspace metadata.

Have you ever lost your eclipse installation and feeling bad that you didn't backup code templates? 

Since code templates are integral part of rapid development a developer should keep his code templates under check. 
In any case if he loses his code templates and have his workspace handy then one can follow below steps to get it back. 

One may wonder where eclipse stores its code templates. Well here is the location. 
C:\Users\<user>\workspace\.metadata\.plugins\org.eclipse.core.runtime\.settings\org.eclipse.php.ui.prefs
Note: The location can change for java and other perspectives. It may be like org.eclipse.java.ui.prefs. I leave the rest to your search within that above folder.

Open that file in your favourite text editor (mine is Notepad++. So screenshots will be in that)
Consider the line(9) where it starts with 

org.eclipse.php.ui.editor.templates


Start copying the part where it looks like above snippet and ends with  

</templates>


Copy the data into another file. 
Personally I don't prefer importing that data directly into eclipse because 
  1. Its eclipse settings file and our required format of xml may differ from raw data of eclipse 
  2. We need to take care of xml safe characters while creating a new xml for code templates 
  3. Eclipse may add other node elements, attributes etc., while generating code templates xml file so please don't try to mimic the functionality of eclipse
Since the raw data we got is not exactly we want so we plan on improving the raw data we got. So lets get started with some simple text manipulations using Regex Expressions. 
First we reorder the data which is earlier in single line.
Note: Don't forget to select Regular expression as your search mode

Match : >< 
Replace : >\r\n< 


After the above operation we can see all templates ordered nicely. (sorry for the dull image)
But there is still a problem with extra backslashes. 
So we remove them

Match : \\=
Replace : =



Add below line at the top of file to make it  as xml with specific encoding
<?xml version="1.0" encoding="UTF-8" standalone="no" ?>
Now save the file as your code templates file and try to import it into eclipse. 


I personally never tried it because I have my eclipse set and I will try it when situation demands.

Please do reply me with your experiences.

Monday, November 7, 2011

Extracting XML data and loading into mysql database

This is my first blog post. I hope this goes well and people understands it easily.

Coming to context we are now going to use a xml to store data into mysql db.

Why XML: Most of the applications out there use xml as the transport medium so someday or other we have to face it.
Tutorial For whom :
  1. This tutorial is for all the people who want an easy way to parse xml, extract data and put the data into database.
  2. It's for the people who migrated from T-SQL and missing the excellent support of xml in mysql.( Atleast I miss it) This will serve as a workaround till the mysql core develops.
Tools needed : MySql database ( I tested it on v 5.5.8), Toad for MySql (optional)
Knowledge required on : extractValue() function of mysql, basic xml knowledge.

NOTE: All the important data will be highlighted in red colour and has courier new font.

Lets start with gaining some basic knowledge of extractValue() function used in MySql.
Well extractValue() as the name suggests it extracts text(cdata) from the xml element.
For more information please have a look at MySql XML Functions Documentation
A basic example of extractValue()
SELECT EXTRACTVALUE('<root><element>data</element></root>','/root/element') AS data;
Make sure that you connect to some database to test and type the above code into your Toad editor window.
You can see the result as below


Now lets decide our xml format. I choose this format to parse.
<params>
    <param>
        <paramname>paramname 1</paramname>
        <paramvalue>paramvalue 1</paramvalue>
    </param>
    <param>
        <paramname>paramname 2</paramname>
        <paramvalue>paramvalue 2</paramvalue>
    </param>
    <param>
        <paramname>paramname 3</paramname>
        <paramvalue>paramvalue 3</paramvalue>
    </param>
</params>
A little bit about xml format :
Here params is the root element under which many param's are present. Each param has a paramname and a paramvalue.

Why this structure :
In general a stored procedure will be having in, out, inout parameters. So if we know what is going in and what is coming out then we can manage inout parameters but think about a case where we don't know how many parameters are needed or else large number of parameters are needed lets say 20. In those cases having 20 parameters in stored procedure is not that easy to maintain for developer and for database caller who sends 20 parameters to it. Instead they both can use a common xml for understanding and parse the xml to get relevant data.In this way it's easy for both ends and it also provides a medium to transport data to one another. This medium can be used when the database is called by more than one application. Lets say a standalone application, web application, mobile application etc., (This is just my thinking . If something is drastically wrong please feel free to contradict ).

To extract the data from xml I wrote a small stored procedure to explain things.
There are 6 steps in doing that

1) Setting up xml as input parameter for stored procedure.

Create a stored procedure something like this
DROP PROCEDURE IF EXISTS newdb1.import_xml;
CREATE PROCEDURE newdb1.`import_xml`(params  TEXT)
BEGIN
END;
Here params is given as text because our xml can grow very long in real time scenarios.

 2) Declaring necessary variables.
DECLARE rowcount INT UNSIGNED DEFAULT 0;
DECLARE paramcount INT UNSIGNED;
DECLARE paramname VARCHAR(255);
DECLARE paramvalue VARCHAR(255);
Here rowcount is to iterate through param's in xml, paramcount is number of param elements in xml, paramname and paramvalue are variables to store extracted data from respective elements.

3) A temporary table to store the extracted data.
CREATE TEMPORARY TABLE temp(paramname VARCHAR(255), paramvalue VARCHAR(2000));
Please note that paramname has given 255 characters and paramvalue is given 2000 because paramname is value of variable name in business context and we don't expect people write stories as variable names and paramvalue is value of variable and here we can expect large data so we gave it 2000. It depends on your business context which length suffices you. Please change as needed.

4) Count the number of params in whole xml.
SET paramcount := EXTRACTVALUE(params,'COUNT(/params/param)');
This will give count of params in xml which in our case is 3.

5) Iterate over each param, extract paramname, paramvalue and inserting them into temp table.
WHILE rowcount < paramcount DO
    SET rowcount := rowcount + 1;
    SET paramname := CONCAT('/params/param[',rowcount,']/paramname[1]');
    SET paramvalue := CONCAT('/params/param[',rowcount,']/paramvalue[1]');
    INSERT INTO temp(paramname, paramvalue) VALUES(EXTRACTVALUE(params, paramname), EXTRACTVALUE(params, paramvalue));
END WHILE;
Here is the interesting part.
A while loop which iterates over all the param elements, extracts data from elements and then it inserts into temp table.

Note : If we use concat('params/param/paramname') then we will get 3 matching elements and if we extract, then all the data in those 3 elements will get concatenated separated by comma which we don't want. So we iterate over elements based on our row count.
Below is the result if we use concat as above


6) Deleting temporary table for the subsequent uses.
DROP TABLE temp;
This deletes the temporary table created and it is a good practice to remove resources when we don't need them anymore. We can have a select statement before deleting to see what data we have got in our table. But in real time scenarios we might not need them.

Note : Please comment all the statements which are for testing, unnecessary select statements and unload all unnecessary resources from memory after completion of task.

Now the final stored procedure with all the bits attached together.
DROP PROCEDURE IF EXISTS newdb1.import_xml;
CREATE PROCEDURE newdb1.`import_xml`(
params                     TEXT
)
/*
'<params>
    <param>
        <paramname>paramname 1</paramname>
        <paramvalue>paramvalue 1</paramvalue>
    </param>
    <param>
        <paramname>paramname 2</paramname>
        <paramvalue>paramvalue 2</paramvalue>
    </param>
    <param>
        <paramname>paramname 3</paramname>
        <paramvalue>paramvalue 3</paramvalue>
    </param>
</params>'
*/

BEGIN
    DECLARE rowcount INT UNSIGNED DEFAULT 0;
    DECLARE paramcount INT UNSIGNED;
    DECLARE paramname VARCHAR(255);
    DECLARE paramvalue VARCHAR(255);

    CREATE TEMPORARY TABLE temp(paramname VARCHAR(255), paramvalue VARCHAR(2000));

    -- calculate the number of row elements.
    SET paramcount := EXTRACTVALUE(params,'COUNT(/params/param)');

    -- loop through all the row elements
    WHILE rowcount < paramcount DO
        SET rowcount := rowcount + 1;
        SET paramname := CONCAT('/params/param[',rowcount,']/paramname[1]');
        SET paramvalue := CONCAT('/params/param[',rowcount,']/paramvalue[1]');
        INSERT INTO temp(paramname, paramvalue) VALUES (EXTRACTVALUE(params, paramname), EXTRACTVALUE(params, paramvalue));
    END WHILE;
    SELECT * FROM temp;
    DROP TABLE temp;
END;
One can call the stored procedure as below. I used concat just to make it visually better for readers but there is no need for that.
CALL import_xml(CONCAT('<params>'
'<param><paramname>paramname 1</paramname><paramvalue>paramvalue 1</paramvalue></param>' 
'<param><paramname>paramname 2</paramname><paramvalue>paramvalue 2</paramvalue></param>'
'<param><paramname>paramname 3</paramname><paramvalue>paramvalue 3</paramvalue></param>'
'</params>') 
);
The final output looks something like this
Since we have the data in temp table we can do anything with it. After this stage we don't use xml to get and set data.