Creating a Community Driven History App

Have you ever wondered what developing a mobile application is like? How difficult is it to start from scratch and deliver something based on a client’s expectations? When we started working on this project, we had 6 weeks to discover and apply new technologies and in the end deliver a working product. While we had prior experience with developing web applications with React and Spring, this was our first time dipping our toes into the realm of native mobile applications.

Our project saw us working on LocalTales, a community-driven history app for Android, which included an Augmented Reality (AR) feature to let users discover the world more immersively, and share their own local knowledge with others in a fun way. In this article we will discuss the technologies we used to create LocalTales, as well as tell you about the challenges we faced and how they were overcome, with some of the solutions being clever, albeit hacky workarounds.

The Tech Stack

Let’s first discuss the tech stack. When looking to create a mobile app with Augmented Reality (AR) capabilities, there are multiple different frameworks that you can choose from. Some of the main ones we looked at were Flutter, Unity and React Native. The first step of our development process was comparing these more thoroughly to find out which one would work best for us.

In the end we decided on the Flutter framework for multiple reasons, including it being developed by Google and our target platform being Android. This meant that the framework would most likely have great support for our target platform, while still remaining cross-platform. The Dart language — which is what Flutter apps are written in — looked similar enough to Java syntax for us to be able to pick it up quickly, while providing modern language features such as a good null-safety system. Other benefits included hot-reloading and restarting capabilities on both the emulator and real devices.

We dismissed Unity early on, because while it offers more well-maintained and advanced AR capabilities, it is not designed for “UI-focused” applications — such as ours — and so other features would have suffered greatly in quality and speed of development as a result. React Native was a comparable alternative to Flutter, but the fact that we already had experience in React was actually one of the arguments against choosing this framework, since this project was focused on us learning new technologies.

Our backend was made in Java using the Spring Boot framework and deployed on Azure with a PostgreSQL database, this was a pretty typical backend setup for us. So to spice things up, for security and authentication instead of our typical choice of Keycloak we went with Firebase. This enabled us to have “Sign in with Google” integrated in our app (which was a requirement from our client). We also integrated Azure Blob Storage with our backend to let users embed media (photos and videos) in their user-generated content using a rich text editor (Quill).

Research Before Application

Before applying anything, it is vital to do some research on possible technologies one could use — their benefits as well as drawbacks. This ensures that you make the right decision at the start of a project. When we started, we had three major questions to answer: what map provider do we use, what AR plugin is best for us to use and how do we provide authentication?

Map Providers

When we started considering the various map providers available to us, we quickly realized there are two major options — Google Maps and OpenStreetMap (OSM). When deciding on which one to choose we had to consider our requirements. Based on what our client suggested, we wanted a solution that was fairly cost effective, very detailed and easy to integrate into our app.

Comparing the two providers, Google Maps charges you per API call which would drive up costs, whereas OSM is a free data service which makes it the more cost effective solution. OSM also has a more detailed and accurate map than Google Maps due to the fact that anyone can easily contribute and improve the OSM data, it is also ad-free. Finally, OSM is designed for you to be able to easily integrate with it, whereas Google Maps may have proprietary connection requirements. The single major drawback for OSM is that there is no native navigation or routing API, but there are some third party routing services that use OSM data as their basis.

AR Plugins

Moving on from the topic of maps, we also had to decide how we would integrate AR into our application, which was one of the major features that our client requested. Since we wanted to have the possibility of having cross-platform Android and iOS applications, we wanted a plugin that could handle both of these. One such plugin for Flutter is the aptly named “ar_flutter_plugin”. Which supports both Google’s ARCore and Apple’s ARKit SDKs, it is also the second most popular AR plugin for Flutter, just behind the iOS-only version it is inspired by.

As you can see, most of the Flutter plugins for AR seem to be mostly or completely abandoned. This caused us issues which we will talk about in the Challenges section.

Authentication Providers

When it came to security of our application and implementation of sign-in methods, we had two options: Keycloak and Firebase Authentication. Since we already had extensive experience with Keycloak, we decided to challenge ourselves and learn about a new platform for us — Firebase. Firebase offers a large number of services ranging from database solutions to authentication. The main way to authenticate users in LocalTales is through Google, though we would consider adding other social media platforms in the future. Additionally, Firebase provides analytics and user insights, if we are to need that in the future.

Implementing Google Sign in with Firebase is rather simple. We started by adding the Firebase Authentication library for Android and the Credential Manager SDK as dependencies in our app. Then we instantiated both Firebase and GoogleSignIn.

Here is a snippet of code we used for the sign in:

Future<User?> signInWithGoogle() async {
 try {
   final GoogleSignInAccount? googleUser = await _googleSignIn.signIn();
   if (googleUser == null) return null;

   final GoogleSignInAuthentication googleAuth = await googleUser.authentication;

   final AuthCredential credential = GoogleAuthProvider.credential(
     accessToken: googleAuth.accessToken,
     idToken: googleAuth.idToken,
   );

   UserCredential userCredential = await _auth.signInWithCredential(credential);

   return userCredential.user;
 } catch (e) {
   debugPrint("Google Sign-In Error: $e");
   return null;
 }
}

Scaling to iOS

When starting the research process, we considered having our application function on both Android and iOS devices. We quickly realised that scaling LocalTales to iOS would require a potentially large amount of time and investment.

First, we would need both a Mac and an iPhone to test, which is something not all of us had access to. Then, to simply test the app on an iPhone, each developer has to sign up for the Developer Program, which costs 99 USD per year, and Apple may take up to multiple weeks to fully approve the account. Setting up the development environment (Xcode) may take from a few hours to a few days depending on the amount of configuration changes required. To test the app locally, we would need to make sure that the Flutter project and configurations for maps and Firebase are updated for iOS.

Due to budget and time constraints, we decided it would be best to leave the scaling to iOS aside. However, all the plugins that we used within our project are compatible with both Android and iOS, so given more time, we would have been able to work out the different required configurations for all the individual plugins and parts of the project.

User Experience

An important aspect when starting to work on a user facing application or project is to ensure that the user experience will be positive and that users will not walk away from our application confused. When looking at user experience (UX), the first aspect that we decided to work on, before anything else, was to create wireframes of what our application would look like. After doing that, we could ask people what they thought of the layout of the application based on just the wireframes without having implemented anything and be able to apply feedback and make changes easily.

Another part of the user experience that we decided to address in our application was the initial introduction of a user to our application. We settled on having a series of onboarding stages to properly show the user around our application after the first time that they download it, so that they could get properly introduced to all the features of our application right away. We also decided to add icons where we could in our application to hopefully help users understand even better what all the various buttons do, as well as positioning most of the key buttons along the bottom of the screen so users could easily find them.

Throughout the development process, we also found it important to ask potential real users of our application to do some UX testing. With this testing in mind we were able to identify and resolve some potential confusion or pain points within our application. One such change was that on our nearby page the rating info point buttons used to be up and down arrow heads which confused our testers. We replaced those with thumbs up and thumbs down icons which are much clearer for all users.

Challenges in Applying Theory

Now that we have researched the technologies that we plan to use, we need to start applying them. Unfortunately, this did not come without its own issues. The main challenge we encountered was with AR, which was a large hurdle for us to overcome.

The first of the issues that we encountered came while trying to run even a simple example AR app with the plugin we selected on the newer Android, Java and Gradle versions. In the end, we could not just use the “official” version of the plugin found on pub.dev, we had to use a more updated fork of a GitHub user @vlad0209. Thankfully Flutter allows for grabbing dependencies straight from a repository:

dependencies:
 ar_flutter_plugin:
   git:
     url: https://github.com/vlad0209/ar_flutter_plugin

Update: There is now a brand new project attempting to revive the ar_flutter_plugin package; it can be found on pub.dev under the creative name of ar_flutter_plugin_2.

Another thing that was breaking the AR was the new Flutter Impeller rendering pipeline, which specifically led to AR not functioning on older Android devices. To combat this, in our AndroidManifest.xml file we had to disable Impeller.

<meta-data
   android:name="io.flutter.embedding.android.EnableImpeller"
   android:value="false" />

After all those issues, you would think that something as simple as displaying text would be a fundamental feature for any self proclaimed AR plugin to have, right? Well, unfortunately, at least in Flutter, this is not the case. The AR plugin we ended up using, ar_flutter_plugin, only supports displaying static 3D models in AR space. However, since we needed to be able to display dynamic data from infopoints in the form of user submitted titles, we needed to find a solution to this, and, I warn you, it is a bit of a hacky workaround.

The first stage for us was to create a 3D model in Blender with a UV texture map of a png image and add it to the project storage. Now that we have that, we had to work out how to display our infopoints titles on it, the first stage for us was to display the text in a widget. Once we had that, we had to take a screenshot of the widget and then save it to local storage. We achieved this with the code below.

At the start of the function we started a new picture recorder and created a new canvas where all the text would be rendered.

final recorder = PictureRecorder();
var newCanvas = Canvas(recorder);

Then once all the text was laid out, we had to take the screenshot of the image and save it.

final picture = recorder.endRecording();
var res = await picture.toImage(width.toInt(), height.toInt());
ByteData? data = await res.toByteData(format: ImageByteFormat.png);
if (data != null) {
 Uint8List uint8list = data.buffer.asUint8List();
 final directory = await getApplicationDocumentsDirectory();
 String filePath = '${directory.path}/$filename.png';
 File imgFile = File(filePath);
 await imgFile.writeAsBytes(uint8list);
 return filePath;
}

The next stage of the workaround involved editing the 3D model that we had created previously to reference this newly created image as its texture as opposed to the placeholder that it had before. After that was done, we then to save the updated 3D model to local storage so that we could use it later.

final gltfString = await rootBundle.loadString(gltfAssetPath);
final Map<String, dynamic> gltfJson = jsonDecode(gltfString);

if (gltfJson.containsKey('images') && gltfJson['images'] is List && gltfJson['images'].isNotEmpty) {
 gltfJson['images'][0]['uri'] = '$filename.png';
 debugPrint('🏔️ Updated image URI to: $newTextureUri');
} else {
 debugPrint('🏔️ No images found in the glTF file.');
}

final updatedGltfString = jsonEncode(gltfJson);

final Directory tempDir = await getApplicationDocumentsDirectory();
final String outputPath = '${tempDir.path}/$filename-model.gltf';
final File outputFile = File(outputPath);
await outputFile.writeAsString(updatedGltfString);

Finally, once all that has been completed, we could use the 3D model to display it correctly in AR space. Quite the workaround that had to happen to get there, I know. You also need to make sure that you have the .bin file for the 3D model that will remain unchanged in the same directory as the modified 3D model and the newly created image.

You can see this workaround illustrated more clearly in the image.

Another challenge that we encountered while working with AR, is that the ARCore sdk is not fully open source which means that the community cannot fix long-standing open GitHub issues such as not being able to test AR in the emulator anymore. To overcome this emulator related issue we had to acquire additional Android phones to be able to properly test the application.

Key Features Integrated

Exploration mode

Since our goal is to make this app effortless and convenient for the user, we designed it to notify them when they’re close to an infopoint. To enable this feature, they would need to grant permission for continuous location tracking and notifications. For handling notifications, we used the “flutter_local_notifications” plugin. To start off, we made sure the dependencies firebase_auth and google_sign_in were set.

To set up location tracking, we needed to set some settings:

LocationSettings locationSettings = LocationSettings(
  accuracy: LocationAccuracy.high,
  distanceFilter: 10,
);

if (defaultTargetPlatform == TargetPlatform.android) {
  locationSettings = AndroidSettings(
    accuracy: LocationAccuracy.high,
    distanceFilter: 10,
    intervalDuration: const Duration(seconds: 10),
    foregroundNotificationConfig: const ForegroundNotificationConfig(
      notificationText: "Tracking your location for nearby info points",
      notificationTitle: "Background Tracking Active",
      enableWakeLock: true,
    )
  );
}

distanceFilter of 10 reduces unnecessary updates while keeping accuracy.

To make sure we do not spam the API, we fetch the infopoints only when necessary, which, as shown in this example, is when the user moves more than 25 meters.

if (lastPosition == null ||
   getDistanceBetween(LatLng(lastPosition!.latitude, lastPosition!.longitude),
       LatLng(position.latitude, position.longitude)) > 25)
{
 try {
   // Creates a bounding box around the user's current position
   final bounds = await fetchSquareBoundsWithinMeters(250, LatLng(position.latitude, position.longitude));
   infoPoints = await fetchInfoPoints(bounds);
   lastPosition = position;
 } catch (e) {
   debugPrint('❌ Error fetching info points: $e');
 }
}

We store infopoints in Set<String> notifiedPoints. The following code block demonstrates how we display the notification to the user.

if (distance <= Config.getPreference<double>("explorationModeDistance")) {
 try {
   await LocalNotificationService.showInstantNotification(
     title: 'Exploration Mode Alert',
     body: 'You are near ${point.title}',
   );
     notifiedPoints.add(point.id);
   if (notifiedPoints.length > 100) {
     notifiedPoints.clear();
   }
 } catch (e) {
   debugPrint("❌ Failed to send notification: $e");
 }
}

To make the application run in the background, we had to make use of a separate plugin flutter_background which we initialize with android settings

final androidConfig = FlutterBackgroundAndroidConfig(
 notificationTitle: "LocalTales Exploration Mode",
 notificationText: "Background notification for keeping LocalTales running in the background",
 notificationImportance: AndroidNotificationImportance.normal,
 notificationIcon: fb.AndroidResource(name: 'background_icon', defType: 'drawable'),
);
bool success = await FlutterBackground.initialize(androidConfig: androidConfig);

And later enable the background execution

await FlutterBackground.enableBackgroundExecution();
LocationService.startLocationListening();

Working With Rich Text

For our application, we wanted to be able to support users formatting the text in the content of our infopoints, using bold, italic and the like. We decided to achieve this by using a rich text editor that used the Quill framework and formatting by using Quill Deltas. Once we settled on a good plugin to use (“flutter_quill” and “flutter_quill_extensions”), it was fairly straightforward to set up our rich text editor to work as expected and add the ability for users to submit their own photos and videos with the info point content as well.

Our rich text editor was configured in the following way. As well as looking like the image that follows the code block.

QuillSimpleToolbar(
     controller: _controller,
     config: QuillSimpleToolbarConfig(
       embedButtons: FlutterQuillEmbeds.toolbarButtons(
         imageButtonOptions: QuillToolbarImageButtonOptions(
           imageButtonConfig: QuillToolbarImageConfig(
             onImageInsertCallback: _uploadImage
           )
         ),
         videoButtonOptions: QuillToolbarVideoButtonOptions(
           videoConfig: QuillToolbarVideoConfig(
             onVideoInsertCallback: _uploadVideo
           )
         )
       ),
       toolbarRunSpacing: 0,
       toolbarIconAlignment: WrapAlignment.start,
       toolbarSectionSpacing: 1,
       showColorButton: false, showCodeBlock: false, showListCheck: false, showInlineCode: false, showSearchButton: false, showFontFamily: false, showListBullets: false, showListNumbers: false, showIndent: false, showBackgroundColorButton: false, showHeaderStyle: false, showSubscript: false, showSuperscript: false, showClearFormat: false, showFontSize: false, showStrikeThrough: false, showLink: false, showQuote: false,
     ),
   ),

Rich Text Editor Result

Moderating User Content

To ensure credibility and avoid misinformation in the infopoints that the user uploads, we have, for now, decided to use a simple web application with an admin dashboard. After a user uploads an infopoint, it is submitted for approval. Admins of our app can either approve this infopoint or deny it, if it is, for example, false information, or contains inappropriate media. At a later point in time, this could be enhanced with AI moderation tools which would enable admins to have all the most misinformative or inappropriate points already filtered out.

Saving Local Preferences

Another feature that we included within our application was having the user be able to highly customize their experience while using the application. To achieve this we needed to store preferences locally on the user’s device, because we aimed to let them use the app without logging in. To store and load these preferences in a convenient way for us, we created a Config class as a wrapper around the shared_preferences Flutter plugin. Since we wanted synchronous accessing of preferences to make using the Config class more convenient, we went with the newer SharedPreferencesWithCache API. We used a generic getter and setter to make accessing and writing with type safety easy, we can also easily register new preferences by just adding them to the defaults map with a correctly typed default value. Below you can find the relevant parts of our Config wrapper class, note that the asynchronous initialize method must be called from main.

static const Map<String, Object?> defaults = {
 "nearbyScreenMaxDistance": 200.0,
 "arShowVoteCountOnInfoPoints": true,
 "onboardingStepsCompleted": 0,
};
static late final SharedPreferencesWithCache _prefs;
static Future<void> initialize() async {
 _prefs = await SharedPreferencesWithCache.create(cacheOptions: SharedPreferencesWithCacheOptions(
   allowList: defaults.keys.toSet()
 ));
 debugPrint("⚙️ Config: Preferences initialized");
}

static bool hasPreference(String key) {
 return defaults.containsKey(key);
}

static void _ensurePreference(String key) {
 if (!hasPreference(key)) throw ArgumentError("Invalid Key: No such preference: $key");
}

static T getPreference<T>(String key) {
 _ensurePreference(key);
 final value = _prefs.get(key) ?? defaults[key];
 if (value is T) return value;
 throw ArgumentError("Cannot retrieve value $value of type ${value.runtimeType} as type $T");
}

static Future<void> setPreference<T>(String key, T value) async {
 _ensurePreference(key);
 if (value == defaults[key]) {
   debugPrint("⚙️ Config: Preference $key reset to default: $value");
   _prefs.remove(key);
   return;
 }

 switch (value) {
   case int v: await _prefs.setInt(key, v);
   case bool v: await _prefs.setBool(key, v);
   case double v: await _prefs.setDouble(key, v);
   case String v: await _prefs.setString(key, v);
   case List<String> v: await _prefs.setStringList(key, v);
   default: throw ArgumentError("Unsupported type: ${value.runtimeType}");
 }
 debugPrint("⚙️ Config: Preference $key set to value: $value");
}

Our App

With all that talk about how we created the app, what about what the app looks like, well below you’ll find some screenshots of what we managed to create.

Conclusion

This project turned out to be a huge learning experience for all three of us. Even though we ran into challenges with the various new technologies, mainly AR, we were able to solve them in time and deliver a good looking, intuitive, well-working Android application for our client.

Looking ahead, there are several ways we’d like to enhance our app. Future improvements would include more extensive UX testing, creating in-app navigation to fully move away from Google Maps, publishing it on the Play Store and expanding support to iOS.

The new things we learned, like the Dart language, Firebase platform, working with AR and fixing the issues with it, improving the UI and simply developing a cross platform app taught us valuable lessons that we will take with us into the start of our career.

We worked on this project as a team of 3 people, Roman Gordon, Boldi Olajos and Anna Vorozhtsova and are very glad with what we managed to deliver.