16
August
2021
Resources

Connect Multiple Data Sources in Your App using these 6 Open Source Tools

0
 minutes

While building an application, there are times when you have to use multiple data sources. It could be for security reasons or availability, or even cost issues. Often, connecting and managing those data sources can be challenging! You either need to build a system from scratch to connect those data sources or use a tool.

Let’s say you’re running an analytics dashboard, and you have data coming from multiple sources like MySQL, Redshift, S3, and MongoDB. Would you want your data scientists to focus on making all API connections, connecting all the data sources, and processing them in a specific format? Or, would you instead let them focus on the other critical parts of their job by using an easy solution and connect those data sources through an app?

Building a complete system from scratch is generally not advisable because of the enormous constraints on time. And, teams also have to go through the documentation for each data source to build a system from scratch. They also have to keep track of everything, such as changelogs/updates in the documentation, and change the code base accordingly. Handling this work would mean hiring resources dedicated to this work or putting the team under pressure to manage all this.  Using an application to oversee the connection makes sense while ensuring that data sources are secure and reliable.

Building everything from scratch can be a time-consuming part that may also distract you from your path.

To help you stay on course, we’ve put together a nifty list of great tools to help you connect multiple data sources without hassles.

Talend

Talend is an open-source data automation tool that can be used to connect multiple data sources. It makes the process of ETL (Extract Transform and Load) pipeline setup more uncomplicated and quicker. It offers a scalable architecture and robust data integration to maximize its value and a suite of open-source tools divided into several components. All combined, they become a potent tool for ETL and connecting multiple data sources.

Talend provides an easy and intuitive way to transform the data. Instead of mapping databases and filling out the forms for different databases, you can just use their graphical tool for mapping and transforming the data.

It also supports data conversion into multiple business formats such as OLAP, Jasper, SPSS, Splunk.

Supported data sources:

  • MSSQL
  • DB2
  • PostGres
  • MySQL
  • Teradata
  • Greenplum
  • DB2
  • Oracle
  • Sybase
  • Vertica
  • Informix

Arcesb

Arcesb is another open-source data integration and pipeline setup tool that can connect multiple data sources. It synchronizes the data in real-time, which means as soon as data gets available in the source, it gets publicly available through the Arcesb.

It supports a wide array of protocols such as AS2, AS4, OFTP, SFTP, etc. The drag-and-drop approach makes it easy to connect complex workflows and quickly transform the data in popular formats like JSON, XML, and CSV.

Supported data sources:

  • MongoDB
  • Dropbox
  • Google Sheets
  • MySQL
  • CSV
  • HubSpot
  • SQLite
  • AWS S3

Apache Camel

Apache Camel is an open-source data integration tool that is a kind of underrated software in ETL. It can be deployed as a standalone application in a web container, connecting complex workflows easily and quickly transformingJEE, OSGi, or even Spring container. It allows programmers to split integration problems into smaller pieces that close the gap between integration and implementation.

It has 3.4k stars on Github and 4.3k Github forks. It supports almost every available protocol like HTTP, HTTP, FTP, JMS, EJB, JPA, RMI, JMS, JMX, LDAP, Netty, and many more. Therefore, it is good to use when you’re dealing with multiple internet protocols. One of the best parts of Apache Camel is that it uses the same workflow for all the supported protocols.

Supported data sources:

  • MySQL
  • ActiveMQ
  • Async HTTP
  • REST API
  • AWS
  • Azure
  • Cassandra CQL
  • MongoDB
  • Digital Ocean
  • FTP/SFTP
  • Google BigQuery
  • GoogleSheets
  • GraphQL
  • RabbitMQ and many more which are listed here

Pentaho Kettle

Pentaho Kettle, also known as the Pentaho Data Integration tool, supports multiple data integrations, OLAP, data mining, reporting, and ETP capabilities. It is also known for its ease to use and quick learning curve.  It allows users to create their data manipulation jobs with a user-friendly graphical creator without entering a single line of code.

It comes with a set of tools that includes:

  • Spoon: This is a graphical tool that lets you create and set up ETL pipelines without using a single line of code. You can perform tasks like validation, refine, transform and write the data to various data destinations.
  • Pan: This is a tool to run data transformations.
  • Chef: This tool is used to create jobs.
  • Kitchen: This tool is used to execute jobs created by Chef.
  • Carte: It is a web server where you can host all your Pans.

Hevo Data

Hevo is a not-code ETL tool that you can use to connect multiple data sources. It supports more than 100 data sources and follows a 3-step data connection setup across all the data sources: select the data source, enter the correct credentials and select the destination.

It uploads the data to the selected destination and allows you to perform data manipulation and transformations. It also comes with a fault-tolerant system that ensures that the data is being transferred in a secured manner consistently using encryption.

Supported data sources:

  • AWS
  • MySQL
  • MongoDB
  • Snowflake
  • TokuDB and many more which you can check here

Appsmith

We have created Appsmith in such a way that you can easily connect multiple data sources with just a few steps. You can even pass the data through an API using our platform. The drag-and-drop UI makes the connection part very easy and allows you to connect data sources like AWS, MySQL, MongoDB, etc.

The credentials that you store in our application are first encrypted before storing. So, you don’t have to worry about security. Since our platform acts as a proxy layer, we do not store any data from your data sources.

To improve the performance, our platform creates a pool of connections with the database server. This tool allows you to run multiple queries as it is impossible to run multiple queries simultaneously in a single link. We have also created in-depth tutorials that you can check out to understand the benefits of other features as well.

Supported data sources:

  • Amazon S3 (also Upcloud, Digital Ocean Spaces, Wasabi, DreamObjects)
  • ArangoDB
  • DynamoDB
  • ElasticSearch
  • Firestore
  • MongoDB
  • MS SQL
  • MySQL
  • PostgreSQL
  • Redis
  • Redshift
  • Snowflake

Are you interested in building something with Appsmith? Take it for a spin! Join our vibrant community on Discord. To get regular updates on what we’re up to, follow us on Twitter.

Connect Multiple Data Sources in Your App using these 6 Open Source Tools

While building an application, there are times when you have to use multiple data sources. It could be for security reasons or availability, or even cost issues. Often, connecting and managing those data sources can be challenging! You either need to build a system from scratch to connect those data sources or use a tool.

Let’s say you’re running an analytics dashboard, and you have data coming from multiple sources like MySQL, Redshift, S3, and MongoDB. Would you want your data scientists to focus on making all API connections, connecting all the data sources, and processing them in a specific format? Or, would you instead let them focus on the other critical parts of their job by using an easy solution and connect those data sources through an app?

Building a complete system from scratch is generally not advisable because of the enormous constraints on time. And, teams also have to go through the documentation for each data source to build a system from scratch. They also have to keep track of everything, such as changelogs/updates in the documentation, and change the code base accordingly. Handling this work would mean hiring resources dedicated to this work or putting the team under pressure to manage all this.  Using an application to oversee the connection makes sense while ensuring that data sources are secure and reliable.

Building everything from scratch can be a time-consuming part that may also distract you from your path.

To help you stay on course, we’ve put together a nifty list of great tools to help you connect multiple data sources without hassles.

Talend

Talend is an open-source data automation tool that can be used to connect multiple data sources. It makes the process of ETL (Extract Transform and Load) pipeline setup more uncomplicated and quicker. It offers a scalable architecture and robust data integration to maximize its value and a suite of open-source tools divided into several components. All combined, they become a potent tool for ETL and connecting multiple data sources.

Talend provides an easy and intuitive way to transform the data. Instead of mapping databases and filling out the forms for different databases, you can just use their graphical tool for mapping and transforming the data.

It also supports data conversion into multiple business formats such as OLAP, Jasper, SPSS, Splunk.

Supported data sources:

  • MSSQL
  • DB2
  • PostGres
  • MySQL
  • Teradata
  • Greenplum
  • DB2
  • Oracle
  • Sybase
  • Vertica
  • Informix

Arcesb

Arcesb is another open-source data integration and pipeline setup tool that can connect multiple data sources. It synchronizes the data in real-time, which means as soon as data gets available in the source, it gets publicly available through the Arcesb.

It supports a wide array of protocols such as AS2, AS4, OFTP, SFTP, etc. The drag-and-drop approach makes it easy to connect complex workflows and quickly transform the data in popular formats like JSON, XML, and CSV.

Supported data sources:

  • MongoDB
  • Dropbox
  • Google Sheets
  • MySQL
  • CSV
  • HubSpot
  • SQLite
  • AWS S3

Apache Camel

Apache Camel is an open-source data integration tool that is a kind of underrated software in ETL. It can be deployed as a standalone application in a web container, connecting complex workflows easily and quickly transformingJEE, OSGi, or even Spring container. It allows programmers to split integration problems into smaller pieces that close the gap between integration and implementation.

It has 3.4k stars on Github and 4.3k Github forks. It supports almost every available protocol like HTTP, HTTP, FTP, JMS, EJB, JPA, RMI, JMS, JMX, LDAP, Netty, and many more. Therefore, it is good to use when you’re dealing with multiple internet protocols. One of the best parts of Apache Camel is that it uses the same workflow for all the supported protocols.

Supported data sources:

  • MySQL
  • ActiveMQ
  • Async HTTP
  • REST API
  • AWS
  • Azure
  • Cassandra CQL
  • MongoDB
  • Digital Ocean
  • FTP/SFTP
  • Google BigQuery
  • GoogleSheets
  • GraphQL
  • RabbitMQ and many more which are listed here

Pentaho Kettle

Pentaho Kettle, also known as the Pentaho Data Integration tool, supports multiple data integrations, OLAP, data mining, reporting, and ETP capabilities. It is also known for its ease to use and quick learning curve.  It allows users to create their data manipulation jobs with a user-friendly graphical creator without entering a single line of code.

It comes with a set of tools that includes:

  • Spoon: This is a graphical tool that lets you create and set up ETL pipelines without using a single line of code. You can perform tasks like validation, refine, transform and write the data to various data destinations.
  • Pan: This is a tool to run data transformations.
  • Chef: This tool is used to create jobs.
  • Kitchen: This tool is used to execute jobs created by Chef.
  • Carte: It is a web server where you can host all your Pans.

Hevo Data

Hevo is a not-code ETL tool that you can use to connect multiple data sources. It supports more than 100 data sources and follows a 3-step data connection setup across all the data sources: select the data source, enter the correct credentials and select the destination.

It uploads the data to the selected destination and allows you to perform data manipulation and transformations. It also comes with a fault-tolerant system that ensures that the data is being transferred in a secured manner consistently using encryption.

Supported data sources:

  • AWS
  • MySQL
  • MongoDB
  • Snowflake
  • TokuDB and many more which you can check here

Appsmith

We have created Appsmith in such a way that you can easily connect multiple data sources with just a few steps. You can even pass the data through an API using our platform. The drag-and-drop UI makes the connection part very easy and allows you to connect data sources like AWS, MySQL, MongoDB, etc.

The credentials that you store in our application are first encrypted before storing. So, you don’t have to worry about security. Since our platform acts as a proxy layer, we do not store any data from your data sources.

To improve the performance, our platform creates a pool of connections with the database server. This tool allows you to run multiple queries as it is impossible to run multiple queries simultaneously in a single link. We have also created in-depth tutorials that you can check out to understand the benefits of other features as well.

Supported data sources:

  • Amazon S3 (also Upcloud, Digital Ocean Spaces, Wasabi, DreamObjects)
  • ArangoDB
  • DynamoDB
  • ElasticSearch
  • Firestore
  • MongoDB
  • MS SQL
  • MySQL
  • PostgreSQL
  • Redis
  • Redshift
  • Snowflake

Are you interested in building something with Appsmith? Take it for a spin! Join our vibrant community on Discord. To get regular updates on what we’re up to, follow us on Twitter.

Square
Try Appsmith
Build a Payment Link Generator App with Stripe APIs
29
April
2022
Resources

Build a Payment Link Generator App with Stripe APIs

Build a Payment Link Generator App with Stripe APIs
Vihar Kurama
0
 minutes ↗
#
stripe
#
guide
#
dashboard
Resources

Stripe is one of the most prominent developer tools for integrating payments into your website or application. The service allows you to start accepting payments from users in 14 countries and 24 currencies, and all this is relatively easy to set up! However, not every business needs a full-fledged website for collecting payments from their customers. In this short tutorial, we'll be building an app on Appsmith that will generate Stripe payment links for you directly from your dashboard. You can create as many payment links as you like and make them available via email. Even if someone doesn't have an Internet connection or uses a computer without a browser installed, they can still take advantage of your services!

Appsmith is an open-source application builder that integrates with custom APIs and databases. It's perfect for building your team's internal tools, admin panels, and dashboards.

Let's dive in!

Setting up Stripe Account

The first step in building a payment link generator is to set up a Stripe account. You can either create a new account or log in if you're an existing user.

Please note that this application is a built-in test mode, which requires additional information about the business to generate payment links. To make it into a fully-functional application, you will need to add additional details regarding your bank and tax information.

Your dashboard will look like this:

CleanShot 2022-04-20 at 15.30.29@2x.png

Even in test mode, you will be able to access all the features of Stripe APIs, but this will not be able to make complete transactions from our generated links.

The next step is to make our API requests from Appsmith; we’ll need to copy the secret key that’s available on the main page of the dashboard.

CleanShot 2022-04-20 at 15.36.24@2x.png

This secret key lets us access our Stripe account via Bearer Token-based authentication.

In the next section, we'll build s simple UI that lets us generate payment links based on the given customer information and payment price.

Building UI on Appsmith

The first step is to create an account on Appsmith. In this guide, I'll be using the cloud version of Appsmith, but you can always choose to use Appsmith locally or self-host it on your server.

  • Navigate to appsmith.com and sign-up for a new account if you're a new user or login into the existing one.
  • Create a new application under your preferred organization. You'll see an editor with everything you need to build your internal application.
  • As soon as you create a new app, you'll see a canvas with all the details around widgets and data sources on the left sidebar.

Now, click on the widgets tab and drag and drop a container widget on the canvas; this will allow us to group all the widgets in a container. This can be completely customizable; you could add borders, background colours, shadows, and more by opening the property pane.

Inside the container widget, drag and drop a new form widget and add a few input widgets onto the container that lets us collect information for payment links:

  • Product Name
  • Price
  • Quantity
  • Success URL
  • Capture Method

We could also add some additional configuration based on the information that needs to be collected, referring to the Stripe Documentation.

Following is a screenshot of how the UI looks on Appsmith:

CleanShot 2022-04-20 at 16.39.52@2x.png

Next, let’s create a new datasource, an API endpoint that’ll create a new Stripe payment link.

  • Click on the + icon next to Datasources from the sidebar
  • Choose API Endpoint and paste the following URL:

https://api.stripe.com/v1/checkout/sessions

  • You can rename this URL by just double-clicking on the existing one; let’s call this stripe-session.
  • Stripe APIs use BEARER token-based authentication; hence, the API expects an Authorization header with a bearer token.
  • Copy the token from the Stripe dashboard and paste it into the header.

Authorization - BEARER <token>

  • Lastly, let’s send the data as a payload using the filling FORM_URLENCODED data since we are collecting all the inputs in a form widget. Alternatively, we could also add the payload in the JSON Body filed.
To bind the data on the API, we’ll need to use the moustache bindings and the input widgets names. Here’s how we can access the data from the price the amount widget:

{{amountInput.text*100}}


Similarly, we add all the required fields on the payload to create a new session. Here’s a screenshot of what the payload looks like:

CleanShot 2022-04-25 at 21.43.03@2x.png

Our API is now ready; let’s add one more input widget, generating a Stripe Session link (the payment link) for use with the data passed through our input widgets.

Here’s what we’ll need to bind the response from the API endpoint; we can do this by binding the following:

​​{{stripe_Session.data.url}}


The .data property on an API request will return the response from the API endpoint; here, we’ve accessed the URL field, which is essentially the payment link.

If you open this URL, you’ll see a new Stripe session with the amount and details you’ve entered on the form.

Here’s a recording of how this works:

If you’re interested in using a database not listed on our website as an integration, please let us know about it by raising a PR on Github, and we will do our best to include it at the earliest.

Join our growing community on Discord, and follow us on Youtube and Twitter to stay up to date.

March Round-up: Templates, JSON Form, and More Product Updates
7
April
2022
Monthly Round-up

March Round-up: Templates, JSON Form, and More Product Updates

March Round-up: Templates, JSON Form, and More Product Updates
Vihar Kurama
0
 minutes ↗
#
announcement
#
applications
#
community
#
Templates
Monthly Round-up

If you’ve followed Appsmith for a while, you know that we can do a lot in 30 days!

I am here to give you all the deets; follow along!

A Headstart for Your Apps!

We’re launching templates! Our ready-made apps are easy to use, forkable, and are bound to give you a little nudge in the right direction. Directly connect your datasource and get customizing!

The Appsmith templates library will be an ever-growing forkable collection of pre-made applications. These apps span across use-cases that will speed up onboarding for new users and makes application development faster.

You can access templates on our website and inside your Appsmith dashboard.

Read the full announcement here.

All-New JSON Form Widget

We’ve just launched the JSON Form Widget, one of our most requested features. It’s live on the cloud app and will be live on the self-hosted app very soon. The JSON form widget is helpful for quickly generating (dynamic or otherwise) forms from JSON fields from queries, APIs, or JS Objects. Check out the documentation for more details.

Here's a short video on how you can generate a form from a table:

Read the full announcement here.

Promises in the Table Buttons

The table widget is one of the most used widgets on Appsmith; it gives the ability to convert column data into different data types, including buttons! With this, developers can add different actions on the onClick property of the button, for example, redirections, showing modals, running queries, etc. Appsmith supports JS throughout the platform using the moustache syntax, but using JS promises to execute actions in the sequence was limited. But now, we got you covered; the Table Button (columns that are set button type) also supports the async-await functions. All triggers are wrapped in a promise, so any missed error will result in an uncaught promise error.

Here’s a simple snippet that can be used in the table button’s onClick property to run a query and then show an alert:


{{
  (function() {
        // the .then will not run if the promise is not returned
        return MockApi.run()
            .then(() => showAlert('success'))
    })()
}}


New JS Powers to Icon Button Widget

Using JavaScript, we can dynamically add and choose icons in the icon button widget. This will allow developers to customize their applications based on conditions, actions, etc. Here’s a simple example: if you’re adding different links to an icon button widget, it can be super handy. You can update the button icon based on the URL – Google Icon when the icon button redirects to a google page, GitHub icon when the icon button redirects to a GitHub page, etc.

CleanShot 2022-03-31 at 11.51.51@2x.png

Not just that, you could also replace the close buttons on the modal with the Icon Button widget for more customization.

Product Updates

Updated Shortcuts for Ease

To create new queries on Appsmith using keyboard shortcuts, you’ll need to use Command/Control + Plus; previously, this was Option + Shift + N. To learn more about all the shortcuts on the platform, you can use Shift + ?.

CleanShot 2022-03-31 at 12.05.33@2x.png
Smart Substitution for known MongoDB Datatypes

Smart substitution handling helps users use MongoDB types like ObjectId or ISODate inside the quotes, and the query works correctly with MongoDB. With this, you can focus more on the logic than worrying about the request data. You can toggle this feature on and off on the query settings page:

CleanShot 2022-03-31 at 12.17.48@2x.png

Following are the MongoDB types that can be handled:

  • ObjectId
  • ISODate
  • NumberLong
  • NumberDecimal
  • Timestamp
Email Notifications on Comments

Several developer teams love the commenting feature on Appsmith when building internal applications together. We've added that the comment author's email is set as the comment notification email to make it more fantastic. You can update these details from the settings page.

CleanShot 2022-03-31 at 12.46.58@2x.png

New Collaborations, Technical Content, Videos & Tutorials

Last month was crazy for us; we’ve published a couple of new blog posts and have successfully hosted four live events!

  • We’ve written an in-depth tutorial on how you can dynamically generate PDF reports from Appsmith using n8n and APISheet. Note that this can also be done by using REST APIs from APISheet.
  • Confidence, our Developer Advocate, made a few guides on using the select widget, adding search for table widget, and using a debugger on Appsmith. Do check them out :)
  • We also created a very cool interview with our engineers Ashok and Rahul on how they’ve built a react library to improve the drag and drop experience on Appsmtth. Watch it here.
  • Not just these, we also pulled out an awesome How Do I Do X on Appsmith (#2) session in our community where we discussed how you could use Pagination, do bulk uploads, and some cool hacks appsmith framework. Watch it here.

If you're curious about what we were up to, look no further and follow this link.

If you’re interested in using a database not listed on our website as an integration, please let us know about it by raising a PR on Github, and we will do our best to include it at the earliest.

Join our growing community on Discord, and follow us on Youtube and Twitter to stay up to date.

Introducing JSON Forms in Appsmith
21
April
2022
Announcement

Introducing JSON Forms in Appsmith

Introducing JSON Forms in Appsmith
Somangshu Goswami
0
 minutes ↗
#
announcement
#
app-development
#
applications
#
widgets
Announcement

Forms are a core part of most internal tools since its one of the major ways in which data is created or updated. Today, we're introducing the JSON Form Widget (documentation), which allows for a fantastic form creation experience and has been one of our top requested features.

The JSON form widget is helpful for quickly generating (dynamic or otherwise) forms from JSON fields.

Here are some highlights of this widget:

Generate Forms from Database Queries, APIs, or JS Objects

Once you drag a new JSON form widget, you will see a default form rendered with some details. You can update the source data field to infer data and then generate the form. This can be bound to any query or variable containing JSON data, such as DB queries, APIs, or JS Objects. The JSON Form widget can be bound to any other widget in Appsmith as well.

Auto Generate Form When Your Data Changes

The form fields are generated according to the source data when you enable an auto-generated form. Fields are generated according to the key-value pairs in the source data. Whenever there is a change in the source data, the form fields get updated automatically.

Configure Fields As You See Fit

Field configuration shows all the fields generated automatically in the forms. You can edit the fields to tweak properties like the field type and default value and bind specific actions by using editable properties. Most fields use the underlying data type-specific widget's properties to allow a full level of customization just like the widget would. For example, if the input type is text input, the editable properties are similar to the input widget in Appsmith.

  • Array Fields allows you to add, remove, and update a group of fields together.
  • Object Fields allow you to group fields together.
  • Add New Field, disable invalid forms (and control them further with JS), and control a widget's visibility on the app page to create highly customized dynamic forms.

Some of the ways that we've seen our users use the JSON form are:

  • Customer service executives select a form template and customize it for a customer. This is stored in a database and then sent over to the customer.
  • Users can create dynamic sign-up forms, as having both the Sign-in and Sign-up forms on the same page. Conditional switching between forms is effortless because you don't need to create two forms; you only need to change the JSON data.
  • Marketing teams are using the JSON form to create personalised form-based campaigns and much more!
Note: It’s live on the cloud app and will be live on the self-hosted version very soon.

Want to explore the JSON Form in detail? Head over to the JSON Form documentation page to learn more.

What’s a Rich Text element?

asdsadasdsa

asdsadasdsa

The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.

sfdfsdfds

dsfdsfdsf

adfkaldf

The rich text element allows you to create and format

sadadasdasdas dsada sadas asd ad

Static and dynamic content editing

  1. vdfgdgd
  2. gjgjg

A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!

How to customize formatting for each rich text

Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.

swzdswxzdsw