Feeds Fetcher - Drupal | DrupalJournal

Both feeds & migrate modules can be used for migrating and importing content in the Drupal system. Both modules have their advantages however we are going to discuss feeds in this article.

Feeds is one of the finest modules in Drupal for importing content. It provides us flexibility and we can extend it easily to full filling our requirements. Some of the best features are

Provides a backend UI for source and field mapping.
We can schedule imports.
Easily configurable, even a non-technical person can configure it from the backend UI.
It allows us to alter content during import, and we can do that from the backend UI itself. (feeds_temper)

It’s one of my favorite modules in Drupal, I have used this module to import XML, JSON, and RSS feeds into the cms and then used paragraphs and views to display them. I have done some most beautiful integration for https://curator.io/ & https://twinesocial.com/ using this module.

We can get more information about the feeds module from its online documentation.

Okay, we are going to develop an SFTP fetcher for the feeds module, which will allow us to fetch JSON files from a specific server directory. It’s based on a requirement I came across a few months ago.

Annotation

<?php
/**
 * Defines a sftp fetcher.
 *
 * @FeedsFetcher(
 *   id = "sftp_fetcher",
 *   title = @Translation("Sftp Fetcher"),
 *   description = @Translation("Use file on a remote server."),
 *   form = {
 *     "configuration" = "Drupal\feeds_sftp_fetcher\Feeds\Fetcher\Form\SftpFetcherForm",
 *     "feed" = "\Drupal\feeds_sftp_fetcher\Feeds\Fetcher\Form\SftpFetcherFeedForm",
 *   },
 * )
 */

id: is the machine name of our fetcher.
title: a translatable title of fetcher.
description: a short translatable description that describes the purpose of the fetcher.
form: includes configuration and feed form
configuration: Configuration form of fetcher and used to set configuration when we are creating a feed type.
feed: Used to set the source of feed, Basically displayed when we are creating a feed content.

Configuration Form

<?php

namespace Drupal\feeds_sftp_fetcher\Feeds\Fetcher\Form;

use Drupal\Component\Utility\Html;
use Drupal\Core\DependencyInjection\ContainerInjectionInterface;
use Drupal\Core\Form\FormStateInterface;
use Drupal\Core\StreamWrapper\StreamWrapperInterface;
use Drupal\Core\StreamWrapper\StreamWrapperManager;
use Drupal\feeds\Plugin\Type\ExternalPluginFormBase;
use Symfony\Component\DependencyInjection\ContainerInterface;

/**
 * Provides a setting form for the SftpFetcher.
 */
class SftpFetcherForm extends ExternalPluginFormBase implements ContainerInjectionInterface {

  /**
   * The stream wrapper manager.
   *
   * @var \Drupal\Core\StreamWrapper\StreamWrapperManager
   */
  protected $streamWrapperManager;

  /**
   * Constructs a DirectoryFetcherForm object.
   *
   * @param \Drupal\Core\StreamWrapper\StreamWrapperManager $streamWrapperManager
   *   The stream wrapper manager.
   */
  public function __construct(StreamWrapperManager $streamWrapperManager) {
    $this->streamWrapperManager = $streamWrapperManager;
  }

  /**
   * {@inheritdoc}
   */
  public static function create(ContainerInterface $container) {
    return new static($container->get('stream_wrapper_manager'));
  }

  /**
   * {@inheritdoc}
   */
  public function buildConfigurationForm(array $form, FormStateInterface $form_state) {
    $form['allowed_extensions'] = [
      '#type' => 'textfield',
      '#title' => $this->t('Allowed file extensions'),
      '#description' => $this->t('Allowed file extensions for upload.'),
      '#default_value' => $this->plugin->getConfiguration('allowed_extensions'),
    ];
    $form['allowed_schemes'] = [
      '#type' => 'checkboxes',
      '#title' => $this->t('Allowed schemes'),
      '#default_value' => $this->plugin->getConfiguration('allowed_schemes'),
      '#options' => $this->getSchemeOptions(),
      '#description' => $this->t('Select the schemes you want to allow for upload.'),
    ];
    $form['host'] = [
      '#type' => 'textfield',
      '#title' => $this->t('FTP/SFTP Host'),
      '#default_value' => $this->plugin->getConfiguration('host'),
      '#required' => TRUE,
      '#description' => $this->t("FTP/SFTP host ip or string."),
    ];
    $form['port'] = [
      '#type' => 'number',
      '#title' => $this->t('FTP/SFTP Port'),
      '#default_value' => $this->plugin->getConfiguration('port'),
      '#required' => TRUE,
      '#description' => $this->t("Numeric port of remote server."),
    ];
    $form['username'] = [
      '#type' => 'textfield',
      '#title' => $this->t('FTP/SFTP Username'),
      '#default_value' => $this->plugin->getConfiguration('username'),
      '#required' => TRUE,
      '#description' => $this->t("FTP/SFTP login user name."),
    ];
    $form['password'] = [
      '#type' => 'textfield',
      '#title' => $this->t('FTP/SFTP Password'),
      '#default_value' => 'CHANGE PASSWORD',
      '#required' => TRUE,
      '#description' => $this->t("FTP/SFTP login password. It will not be shown here."),
    ];
    return $form;
  }

  /**
   * {@inheritdoc}
   */
  public function validateConfigurationForm(array &$form, FormStateInterface $form_state) {
    $form_state->setValue('allowed_schemes', array_filter($form_state->getValue('allowed_schemes', [])));

    $extensions = preg_replace('/\s+/', ' ', trim($form_state->getValue('allowed_extensions', '')));
    $form_state->setValue('allowed_extensions', $extensions);
  }

  /**
   * {@inheritdoc}
   */
  public function submitConfigurationForm(array &$form, FormStateInterface $form_state) {
    // Keep password same.
    $password = $form_state->getValue('password');
    if (empty($password) || $password == 'CHANGE PASSWORD') {
      $form_state->setValue('password', $this->plugin->getConfiguration('password'));
    }
  }

  /**
   * Returns available scheme options for use in checkboxes or select list.
   *
   * @return array
   *   The available scheme array keyed scheme => description.
   */
  protected function getSchemeOptions() {
    $options = [];
    foreach ($this->streamWrapperManager->getDescriptions(StreamWrapperInterface::WRITE_VISIBLE) as $scheme => $description) {
      $options[$scheme] = Html::escape($scheme . ': ' . $description);
    }

    return $options;
  }

}

Feed form

<?php

namespace Drupal\feeds_sftp_fetcher\Feeds\Fetcher\Form;

use Drupal\Core\Form\FormStateInterface;
use Drupal\feeds\FeedInterface;
use Drupal\feeds\Plugin\Type\ExternalPluginFormBase;
use Drupal\feeds\Utility\File;

/**
 * Provides a form on the feed edit page for the SftpFetcher.
 */
class SftpFetcherFeedForm extends ExternalPluginFormBase {

  /**
   * {@inheritdoc}
   */
  public function buildConfigurationForm(array $form, FormStateInterface $form_state, FeedInterface $feed = NULL): array {
    $args = ['%schemes' => implode(', ', $this->plugin->getConfiguration('allowed_schemes'))];

    $form['source'] = [
      '#title' => $this->t('Remote server file path'),
      '#type' => 'textfield',
      '#default_value' => $feed->getSource(),
      '#allowed_schemes' => $this->plugin->getConfiguration('allowed_schemes'),
      '#description' => $this->t('The allowed schemes are: %schemes, You can also use token here.', $args),
    ];

    return $form;
  }

  /**
   * {@inheritdoc}
   */
  public function validateConfigurationForm(array &$form, FormStateInterface $form_state, FeedInterface $feed = NULL) {
    $source = $form_state->getValue('source');

    $allowed = $this->plugin->getConfiguration('allowed_extensions');

    // Validate a single file.
    if (!File::validateExtension($source, $allowed)) {
      $form_state->setError($form['source'], $this->t('%source has an invalid file extension.', ['%source' => $source]));
    }
  }

  /**
   * {@inheritdoc}
   */
  public function submitConfigurationForm(array &$form, FormStateInterface $form_state, FeedInterface $feed = NULL) {
    $feed->setSource($form_state->getValue('source'));
  }

}

Fetcher Plugin

<?php

namespace Drupal\feeds_sftp_fetcher\Feeds\Fetcher;

use Drupal\Core\File\FileSystemInterface;
use Drupal\Core\Logger\LoggerChannelFactoryInterface;
use Drupal\Core\Plugin\ContainerFactoryPluginInterface;
use Drupal\Core\Utility\Token;
use Drupal\feeds\Exception\EmptyFeedException;
use Drupal\feeds\FeedInterface;
use Drupal\feeds\Plugin\Type\Fetcher\FetcherInterface;
use Drupal\feeds\Plugin\Type\PluginBase;
use Drupal\feeds\Result\FetcherResult;
use Drupal\feeds\StateInterface;
use Drupal\feeds\Utility\File;
use phpseclib3\Net\SFTP;
use Symfony\Component\DependencyInjection\ContainerInterface;

/**
 * Defines a sftp fetcher.
 *
 * @FeedsFetcher(
 *   id = "sftp_fetcher",
 *   title = @Translation("Sftp Fetcher"),
 *   description = @Translation("Use file on a remote server."),
 *   form = {
 *     "configuration" =
 *   "Drupal\feeds_sftp_fetcher\Feeds\Fetcher\Form\SftpFetcherForm",
 *     "feed" =
 *   "\Drupal\feeds_sftp_fetcher\Feeds\Fetcher\Form\SftpFetcherFeedForm",
 *   },
 * )
 */
class SftpFetcher extends PluginBase implements FetcherInterface, ContainerFactoryPluginInterface {

  /**
   * File system object.
   *
   * @var \Drupal\Core\File\FileSystemInterface
   */
  protected $fileSystem;

  /**
   * Logger channel factory object.
   *
   * @var \Drupal\Core\Logger\LoggerChannelInterface
   */
  protected $logger;

  /**
   * Drupal token.
   *
   * @var \Drupal\Core\Utility\Token
   */
  protected $token;

  /**
   * Constructs an UploadFetcher object.
   *
   * @param array $configuration
   *   The plugin configuration.
   * @param string $plugin_id
   *   The plugin id.
   * @param array $plugin_definition
   *   The plugin definition.
   * @param \Drupal\Core\File\FileSystemInterface $fileSystem
   *   File system service.
   * @param \Drupal\Core\Logger\LoggerChannelFactoryInterface $loggerChannelFactory
   *   Logger service.
   * @param \Drupal\Core\Utility\Token $token
   *   Token replacement utility.
   */
  public function __construct(array $configuration, $plugin_id, array $plugin_definition,
                              FileSystemInterface $fileSystem,
                              LoggerChannelFactoryInterface $loggerChannelFactory,
                              Token $token) {
    $this->fileSystem = $fileSystem;
    $this->logger = $loggerChannelFactory->get('feeds_sftp_fetcher');
    $this->token = $token;
    parent::__construct($configuration, $plugin_id, $plugin_definition);
  }

  /**
   * {@inheritdoc}
   */
  public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition) {
    return new static(
      $configuration,
      $plugin_id,
      $plugin_definition,
      $container->get('file_system'),
      $container->get('logger.factory'),
      $container->get('token')
    );
  }

  /**
   * {@inheritdoc}
   */
  public function fetch(FeedInterface $feed, StateInterface $state) {

    $path = $this->getFileFromSftp($feed, $state);

    // Just return a file fetcher result if this is a file. Make sure to
    // re-validate the file extension in case the feed type settings have
    // changed.
    if (is_file($path)) {
      if (File::validateExtension($path, $this->configuration['allowed_extensions'])) {
        return new FetcherResult($path);
      }
      else {
        throw new \RuntimeException($this->t('%source has an invalid file extension.', ['%source' => $path]));
      }
    }

    throw new EmptyFeedException();
  }

  /**
   * {@inheritdoc}
   */
  public function defaultFeedConfiguration() {
    return ['source' => ''];
  }

  /**
   * {@inheritdoc}
   */
  public function defaultConfiguration() {
    return [
      'allowed_extensions' => 'csv xml json',
      'allowed_schemes' => ['public'],
      'host' => NULL,
      'port' => '22',
      'username' => NULL,
      'password' => NULL,
    ];
  }

  /**
   * Fetch file from sftp.
   * @param \Drupal\feeds\FeedInterface $feed
   * @param \Drupal\feeds\StateInterface $state
   *
   * @return string|null
   */
  public function getFileFromSftp(FeedInterface $feed, StateInterface $state) {
    // Source.
    $path = $feed->getSource();

    // Replace token
    $path = $this->token->replace($path);

    // Separate file name.
    $file_name = pathinfo($path, PATHINFO_BASENAME);
    $server_path = pathinfo($path, PATHINFO_DIRNAME);

    // Check compatibility and Prepare download directory
    $downloadDir = 'public://sftp_sync';
    if (!$this->fileSystem->prepareDirectory($downloadDir, FileSystemInterface::CREATE_DIRECTORY)
    ) {
      $this->logger->notice('Download directory is not ready');
      return NULL;
    }

    // Connecting to sftp.
    $this->sftp = new SFTP(
      $this->configuration['host'],
      $this->configuration['port']
    );

    // Connect to sftp
    if (!$this->sftp->login($this->configuration['username'], $this->configuration['password'])) {
      $this->logger->notice('SFTP login failed');
      return NULL;
    }

    // Change dir
    $this->sftp->chdir($server_path);

    try {
      // Get data from sftp
      $data = $this->sftp->get(pathinfo($path, PATHINFO_BASENAME));

      // Destination file path.
      $_file = $downloadDir . '/' . $file_name;

      // Save data to drupal file system
      $this->fileSystem->saveData(
        $data,
        $_file,
        FileSystemInterface::EXISTS_REPLACE
      );

      // Return file path.
      return $_file;

    }
    catch (\Exception $e) {
      $this->logger->notice($e->getMessage());
    }

    return NULL;
  }

}

Note: PS Sftp vendor is used to establish an SFTP connection.

I assume that we hardly get such requirements because most of the sources are available using web URLs but a feeds fetcher can be developed when none of the available fetchers is working for us.

The complete code of this module is available here — https://www.drupal.org/project/feeds_sftp_fetcher