Creating a Flink Stream Processing Project
Note: You can skip this step if you have a streaming project set up already.
Please use the following project templates and setup guidelines, to set up a stream processing project with Apache Flink using Connectors
Once after the set up, please follow the below instructions to add the Flink Pravega connectors to the project.
Add the Connector Dependencies
To add the Pravega connector dependencies to your project, add the following entry to your project file: (For example,
pom.xml for Maven)
<!-- Before Pravega 0.6 --> <dependency> <groupId>io.pravega</groupId> <artifactId>pravega-connectors-flink_2.12</artifactId> <version>0.5.1</version> </dependency> <!-- Pravega 0.6 and After --> <dependency> <groupId>io.pravega</groupId> <artifactId>pravega-connectors-flink-1.9_2.12</artifactId> <version>0.6.0</version> </dependency>
Use appropriate version as necessary.
1.9 is the Flink Major-Minor version.
2.12 is the Scala version.
0.6.0 is the Pravega version.
The snapshot versions are published to
jcenter repository and the release artifacts are available in
Maven Central repository.
Alternatively, we could build and publish the connector artifacts to local maven repository by executing the following command and make use of that version as your application dependency.
./gradlew clean install
Running / Deploying the Application
From Flink's perspective, the connector to Pravega is part of the streaming application (not part of Flink's core runtime), so the connector code must be part of the application's code artifact (JAR file). Typically, a Flink application is bundled as a
fat-jar (also known as an
uber-jar) , such that all its dependencies are embedded.
The project set up should have been a success, if you have used the above linked templates/guides.
If you set up a application's project and dependencies manually, you need to make sure that it builds a jar with dependencies, to include both the application and the connector classes.
The Flink connector has embedded and shaded the
pravega-clientdependency of the same version. Please DO NOT include both
pravega-flink-connectordependency into the jar file, otherwise there will be dependency conflict issues.
If user application uses Table API and SQL, please DO NOT compile
flink-table-planner-blinkinto the jar file as they are provided by the Flink cluster.