It comes as no surprise that modern business should effectively deal with all sorts of information. We will be able to help you out with every aspect of data mamagement.
This is the amount of information created within your company, your personal property and a starting point to your success. What you would certainly choose to keep private are:
- Product and services sales figures. These may also include data from e-commerce or accounting systems, for instance, product volume and product segmentation in accordance with product categories, points of sales and time of day/week/year.
- Information available through data analytics systems on the company’s website or coming from external trading platforms, stored in Google Analytics, Yandex.Metrics and the like. It may also include information on site visitors, their segmentation and interaction with the website, such as duration of each visit, cancellations, events, and much more.
- Reference materials, such as articles, manuals, videos, photos, market reviews, and other materials employed by your company for a more efficient performance.
- Personnel records, such as contact details, personal information, and integration with sales department.
This includes any data sources beyond your control, such as:
- Advertising tools, among which are contextual advertising systems (Yandex.Direct, Google Ads), advertising in social networks, ads posted at websites that correlate with your products. It also includes a variety of internet advertising models, among which are CPC (cost per click), CPI (cost per impression), and CPA (cost per action), all closely integrated with analytics systems.
- Social networks providing users’ personal data, interests, and activity statistics of your social network groups, competitors’ accounts, and other online communities.
- Integration with your partners’ data which may include sales figures at partners’ electronic trading platforms, product export and import, sales analytics, and analysis of advertising campaign performance.
- Competitors’ information gathered through spying on competitors and obtaining data on their products, their rotation, price changes, marketing tools, internal information.
- Market analysis including changes in demand, projected growth and market development, major events in industries.
- Company databases in multiple industries for B2B sales. These include specialized data systems and open directories, such as Yelp, LinkedIn, Yellow Pages and other business directory databases.
Now, how do we restore order in your data kingdom?
Here are a few tools our team has developed to obtain data from various sources:
- Simple crawlers are basic software modules that look for simple flat data located, for instance, at static pages of websites that do not require authorization or search form filling. These crawlers boast amazing performance but they can not be automated easily, nor can they deal with Ajax requests. They are also blocked by most anti-bot systems.
- Advanced crawlers are a more efficient version of crawlers allowing for form filling, cookies acceptance and HTTP POST requesting. They can be easily automated and are highly productive. In the meantime they are unable to deal with Ajax requests or pass through anti-bot systems.
- Virtual users that fully resemble real-life users but are created by a full-featured Chrome or FireFox engine. With virtual users, that are almost impossible to distinguish from real-life users, you have no restrictions when it comes down to form filling, Ajax requests or more elaborate user interfaces.
- Hacking anti-bot systems with premium proxy servers, balancing activities of virtual users, account rotation, employing multiple profiles of embedded browsers with system data rotation (operating systems, languages, resolutions, etc.)
It’s always easier to get understanding with few samples.
Automated product placement
An online seller of kitchenware with about 1,000 goods in the stock list asked for an opportunity to automate product placement at its own website, with product information (such as names, prices, available quantities, descriptions, specifications, and photos) to be posted automatically at Amazon and eBay.
- For the client’s site based on WordPress and wooCommerce we came up with an extra module that helped forward products directly to Amazon and eBay online stores.
- The stores’ APIs were used to access sales figures and automate order processing.
- All sales figures were integrated into a unified accounting system.
«These HandyData guys’re amazing! It took them about a month to automate order processing at Amazon and eBay which saved us loads of time and yup, money too. Thanks, guys, you saved us about 20-30K a year and that’s at a pretty conservative estimate. Keep up the good work!» – Micheal L., N.Y., USA.
Automated tracking of website position in Google search results
The search engine optimization (SEO) company wished to automate the process of tracking Google positions for hundreds of sites with thousands of keywords.
- We developed a frontend system for creating all the necessary entities (websites, keywords, positions at any given time).
- We created a backend web service to automate collecting position data.
- We implemented a data gathering system based on virtual users technology.
- We worked out scripts that simulate real-life users based on a variety of scenarios and technical characteristics
«We do have a bunch of software developers but we realized the complexity of the task and opted for delegation. There were times I had doubts they would be able to make it but the HandyData team came out with flying colors. I was truly surprised they figured it out since Google is known to hate bots and block suspicious IP addresses and even subnets. Well, I am glad to find I have been mistaken as the system created is up and running and has already improved our SEO services to a significant extent.» – Susana B., California, USA.
Scraping tools examples
Below you will find a list of scrapers we built as a quick demo of our skills: