Find unique elements
Unique CSS selectors select only one element. This is useful, for example, for finding titles and featured images.
WP Content Crawler is a WordPress plugin that helps you get content from other websites, into your website.
Now with ChatGPT!Unique CSS selectors select only one element. This is useful, for example, for finding titles and featured images.
The "similar" mode finds a CSS selector that selects the items similar to the clicked item. This is useful, for example, for finding post tags and product attributes.
The "contains" mode finds a CSS selector that selects the element containing all the clicked elements. This is useful, for example, for finding post content and excerpt.
Find and replace anything in the target page. You can also use regular expressions. You can use dedicated options to find and replace in the post title, content, tags, post meta, and others…
Learn more about finding and replacingYou can remove and clone elements, remove and exchange element attributes, find and replace in element HTML and attributes, remove empty elements, remove links by domain…
Learn more about manipulating HTMLMake text upper case, lower case, title case, camel case, snake case, kebab case, and more… You can also limit the length by character count or word count.
Learn more about advanced text manipulationsUse the magnifier button to quickly test the configurations. You can test find-replace options, CSS selectors, formulas, filter commands, and more… You can also enable caching for the quick tests to make them even faster.
The Site Tester page helps you check if your configurations work as expected. You just enter a category or post page URL and hit the test button. The plugin crawls the page and shows you all the details it finds in the page, along with some performance information and, if any, warnings.
Learn more about testing the settingsAutomatically transform, translate, and spin specific parts of the posts
You can transform the parts of the post by using paid OpenAI GPT models, like ChatGPT. Use the cutting-edge AI models to generate SEO-friendly summaries, insert interesting facts, rewrite posts, fix grammar errors, and do anything else that you can imagine.
Learn more about transformationYou can automatically translate the posts from one language to another by using one of the following paid services. You can also choose which parts of the post should be translated.
Learn more about translatingSpinning is like paraphrasing, keeping the same meaning while using different words. You can automatically spin the posts by using one of the following paid services. You can also choose which parts of the post should be spun.
Learn more about spinningThe filter above assigns the status of the post as "draft" if the crawled post's title has less than 20 characters. Using many condition commands available in the plugin, you can create complex conditions. You can combine the conditions with "and" and "or" operators as well. When the condition is met, you can do multiple things, such as sending an email notification to yourself, adding the post to a specific category, finding and replacing things, assigning a specific featured image, calculating a value, and many more.
Use the options dedicated to WooCommerce products to easily save the product details
The WooCommerce section's design is similar to the design of the original WooCommerce options. Here, instead of entering the values, you define CSS selectors. You can create simple and external products. You can also mark the product as virtual and/or downloadable. You can retrieve the downloadable file from the target page as well.
Learn more about WooCommerce optionsSave the crawled posts as custom post types instead of regular WordPress posts
Select the type of post you want to create, if you do not want regular WordPress posts. All the post types registered in your WordPress site are available. You can select a different post type for each site or set the post type for all the sites from the general settings.
Learn more about saving custom post typesCustom post types typically keep extra information in their custom fields. You can save the values of the custom fields in plain text format so that the custom post type can use them accordingly.
Learn more about saving custom fieldsCustom post types might also keep extra information in specific taxonomies, such as tags, company name, ISBN number, and so on. You can save the values of the taxonomies in plain text format so that the custom post type can use those values in the correct places.
Learn more about saving taxonomy valuesCustom post types might have their own category taxonomy. You can save the posts into those categories by defining the category taxonomy in the general settings. Then, you can select those categories in the Category URLs setting. The rest is handled by the plugin.
Learn more about using custom post type categoriesSee the active sites, when they were crawled the latest, how many posts were crawled, how many URLs are in the queue, and more…
Learn more about the active sitesSee when the URL collection, crawling, deleting, and recrawling events will be executed and for which site. You can also see the total number of posts in queue, crawled, recrawled, and deleted. While a post is being crawled or recrawled, it is also displayed in this section.
Learn more about seeing what's happeningSee the latest posts that were crawled, recrawled, and deleted. You can also make the dashboard page refresh in specific time intervals to follow the events as they happen.
Learn more about recently crawled postsThe options box is displayed by clicking to the "settings" button, shown in the image with a blue border. The button is available for many settings that define a CSS selector.
Learn more about the options boxYou can define one or many proxy servers. While crawling, the plugin tries them one by one until it reaches the target website. The plugin uses these proxies every time it makes a request to the target site.
Learn more about using proxiesYou can make the plugin randomly use a proxy from the list of proxies every time it makes a request to the target site.
Learn more about randomizing proxiesDefine CSS selectors that must always find a value and then get an email when they do not have a value.
Learn more about empty value notificationsBy using the filters, you can define your custom conditions and send yourself a custom email when those conditions are met. You can include the URL of the post, HTTP status code of the response, name, ID and edit URL of the site, and current time in your email. You can also leave the email template empty to use the default template provided by the plugin.
Learn more about custom notificationsAt the bottom right corner of the plugin's pages, there is a "Guides" button. When you click it, you see the list of interactive guides available in the plugin. From there, you can search for a guide and click its "play" button to start it. You can also start a guide from a specific step.
Assign cookies to each request to crawl as a logged-in user
Add request headers to each request
Do some standard operations by simply checking checkboxes
Check for duplicate posts via URL, title, content
Add link to the source or anything to the post content
Assign an element to a shortcode and use it in the templates
Fill the category URLs setting automatically
Define password for the crawled posts
Change the general settings per site
Save posts as draft and check before publishing
Assign the crawled posts to a custom user
Save all images in the post content by checking a checkbox
Change the publish date to a future date
Save the changes in a few milliseconds
Convert the encoding to UTF-8 easily
Easily clone or move the site settings to another website
Check out all the guides