Quantcast
Channel: adrianwalker.org
Viewing all 65 articles
Browse latest View live

HTML5 Video Pseudostreaming with Java 7

$
0
0

Adapted from the byte range request servlet code from the The BalusC Code, here is a Java 7 pseudostreaming servlet for playing video using the HTML5 video tag.

The servlet takes the name of a video file to play as a request parameter and the byte range of the video to stream in the Range header field.

PseudostreamingServlet.java

package org.adrianwalker.pseudostreaming;

import java.io.IOException;
import java.io.OutputStream;
import java.net.URLDecoder;
import java.nio.ByteBuffer;
import java.nio.channels.SeekableByteChannel;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import static java.nio.file.StandardOpenOption.READ;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

public final class PseudostreamingServlet extends HttpServlet {

  private static final int BUFFER_LENGTH = 1024 * 16;
  private static final long EXPIRE_TIME = 1000 * 60 * 60 * 24;
  private static final Pattern RANGE_PATTERN = Pattern.compile("bytes=(?<start>\\d*)-(?<end>\\d*)");
  private String videoPath;

  @Override
  public void init() throws ServletException {
    videoPath = getInitParameter("videoPath");
  }

  @Override
  protected void doGet(final HttpServletRequest request, final HttpServletResponse response) throws ServletException, IOException {
    processRequest(request, response);
  }

  private void processRequest(final HttpServletRequest request, final HttpServletResponse response) throws IOException {

    String videoFilename = URLDecoder.decode(request.getParameter("video"), "UTF-8");
    Path video = Paths.get(videoPath, videoFilename);

    int length = (int) Files.size(video);
    int start = 0;
    int end = length - 1;

    String range = request.getHeader("Range");
    Matcher matcher = RANGE_PATTERN.matcher(range);
    
    if (matcher.matches()) {
      String startGroup = matcher.group("start");
      start = startGroup.isEmpty() ? start : Integer.valueOf(startGroup);
      start = start < 0 ? 0 : start;

      String endGroup = matcher.group("end");
      end = endGroup.isEmpty() ? end : Integer.valueOf(endGroup);
      end = end > length - 1 ? length - 1 : end;
    }

    int contentLength = end - start + 1;

    response.reset();
    response.setBufferSize(BUFFER_LENGTH);
    response.setHeader("Content-Disposition", String.format("inline;filename=\"%s\"", videoFilename));
    response.setHeader("Accept-Ranges", "bytes");
    response.setDateHeader("Last-Modified", Files.getLastModifiedTime(video).toMillis());
    response.setDateHeader("Expires", System.currentTimeMillis() + EXPIRE_TIME);
    response.setContentType(Files.probeContentType(video));
    response.setHeader("Content-Range", String.format("bytes %s-%s/%s", start, end, length));
    response.setHeader("Content-Length", String.format("%s", contentLength));
    response.setStatus(HttpServletResponse.SC_PARTIAL_CONTENT);

    int bytesRead;
    int bytesLeft = contentLength;
    ByteBuffer buffer = ByteBuffer.allocate(BUFFER_LENGTH);

    try (SeekableByteChannel input = Files.newByteChannel(video, READ);
            OutputStream output = response.getOutputStream()) {

      input.position(start);

      while ((bytesRead = input.read(buffer)) != -1 && bytesLeft > 0) {
        buffer.clear();
        output.write(buffer.array(), 0, bytesLeft < bytesRead ? bytesLeft : bytesRead);
        bytesLeft -= bytesRead;
      }
    }
  }
}

The location of the videos on the file system is configurable in the web.xml via the videoPath parameter for the stream servlet.

web.xml

<?xml version="1.0" encoding="UTF-8"?><web-app version="3.0" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"><servlet><servlet-name>stream</servlet-name><servlet-class>org.adrianwalker.pseudostreaming.PseudostreamingServlet</servlet-class><init-param><param-name>videoPath</param-name><param-value>/tmp/videos</param-value></init-param></servlet><servlet-mapping><servlet-name>stream</servlet-name><url-pattern>/stream</url-pattern></servlet-mapping><session-config><session-timeout>
            30
        </session-timeout></session-config></web-app>

Some simple HTML5 uses the servlet as the source for the video tag. The name of the video files is provided by the video URL parameter.

index.html

<!DOCTYPE html><html><head><title>HTML5 Video Pseudostreaming</title><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body><div><video id="video" controls><source src="stream?video=video.webm" /></video></div></body></html>

You're going to need a video to play, so download a trailer or something:

http://trailers.apple.com/movies/sony_pictures/theamazingspiderman/amazingspiderman-tlr2b-USA_h480p.mov

The QuickTime trailer video file isn't supported by my target browser, so I need to convert it to another format, for this example WebM. ffmpeg can be used for the conversion:

ffmpeg -i amazingspiderman-tlr2b-USA_h480p.mov /tmp/videos/video.webm

Source Code

Usage

Run the project with 'mvn clean install jetty:run-war' and point your brower at http://localhost:8080/pseudostreaming.


Java EE 6 Cookbook for Securing, Tuning, and Extending Enterprise Applications Review

$
0
0

Java Enterprise Edition is Oracle's enterprise Java computing platform. The platform provides an API and runtime environment for developing and running enterprise software, including network and web services. Java EE extends the Java Standard Edition providing an API for object-relational mapping, distributed and multi-tier architectures, and web services.

Packt Publishing requested that I review one of their titles about Java EE: Java EE 6 Cookbook for Securing, Tuning, and Extending Enterprise Applications by Mick Knutson, available to buy from Packt's website.

Java EE 6 Cookbook for Securing, Tuning, and Extending Enterprise Applications is an OK book, the breadth of topics covered is vast, but outside of the scope of the book at times. It feels like the book is trying to be an introductory tutorial to most aspects of Java development. Not that this is a bad thing, lots of the topics covered are important to day to day Java development and the advice given is pragmatic. Using the information as a starting point to expand your knowledge would make you a Java jack of all trades.

I found parts of it very informative and other parts frustrating, overall I'd say its worth a read but might not be what you expect. It not a book about JEE development as much as about peripheral concerns around JEE development which you might encounter in a project.

The information is presented in recipes - how-to examples focusing on a specific JEE feature, or other topic, put together to form a cookbook. Each recipe is presented in a formulaic way, with information under the sub-headings: 'Getting ready', 'How to do it...', 'How it works...' and 'There's more'.

A Java EE 6 installation, application server, and an editor are requirements to reproduce the examples in the book. GlassFish application server is used in a number of recipes, and some recipes are specific to the IntelliJ IDEA editor, but for the majority of the book any IDE and app server could be used.

The code to accompany the book available as a download isn't great. The Maven project pom.xml files needed altering before the code would build, and source code examples for some chapters seemed to be altogether missing.

Chapter one highlights key changes in Java EE 6 and how new features will improve performance and ease of development. The chapter first covers old JEE APIs, showing examples of what older technologies in old applications might need replacing, and what to replace those older technologies with. Covered next are introductions to the specifications and APIs including, Context and Dependency Injection (CDI), EJB 3.1, JPA 2.0, JAX-RS 1.1, Sevlet 3.0, WebBeans 1.0, and JSF 2.0.

Chapter two covers JPA persistence, some good recipes in this chapter are the 'Understanding @CollectionTable' recipe, covering the new annotation in the JPA 2.0 specification, and the JPA audit pattern in recipe 'Auditing historical JPA Operations'. The chapter finishes with an introduction to profiling JPA operations with the IntelliJ IDEA profile. I've never gotten on board with IntelliJ IDEA, and much prefer NetBeans and its profiler.

Chapter three covers JEE security, starting with definitions of security terms and deployment descriptors for role definitions and mapping in GlassFish. Next is a pointlessly detailed description of Java EE authentication methods complete with activity diagrams and HTML request/response headers.

XML based authorization configuration is introduced, then new programmatic and annotation based security in the next recipe.

A recipe I found useful later in the chapter was 'Configuring Linux firewall rules' - using iptables to configure access to SSH on Linux. Although I found it informative, I can't help thinking it doesn't really belong in this book - and the same with the next recipe 'Securely obfuscating Java byte-code', I fail to see how obfuscating code makes an application any more secure.

Chapter 4 deals with testing. Testing is clearly an important topic, and consideration of testing strategies, frameworks, tools and libraries should part of any Java project, but I was surprised to see this as a chapter in this book. It's more of a general software development topic which probably needs a book in its own right, I'm not sure it should be in a book about Securing, Tuning and Extending Enterprise Java apps.

The chapter starts with a good recipe demonstrating the use of the Maven Surefire plugin to enable a remote debugging session.

The rest of the chapter gives examples of testing frameworks and libraries, not just applicable to JEE testing but also to Java SE. JUnit and DBUnit are used automate database testing, along with test object mocking libraries Mockito, Powermock and Hamcrest.

Selenium and soapUI testing tools are introduced at the end of the chapter, with some good information on how to integrate these tools into your build and test process with Maven.

Chapter 5 uses the Groovy and Scala scripting languages running on the JVM, integrated with existing Java code, to write more concise unit test code. There is also a Jython servlet recipe demonstrating how to create a simple web page. Although not strictly JEE related, I enjoyed these recipes and can see how using them to write less code can have productivity increases.

The rest of the chapter deals with Aspect Orient Programming with AspectJ and CDI, and starts by describing some AOP terms: Cross-cutting-concerns, Advice, Pointcut, Aspect Joinpoint and Weaving. Recipes include weaving AspectJ advice into your own code and using AspectJ with existing 3rd party libraries, and using CDI interceptors. The AspectJ recipes in this chapter are not limited to JEE programming, and would be applicable to Java SE projects also.

Chapter 6 goes a totally off track with mobile development recipes. Recipes cover mobile web frameworks, native code generators, native web runtime frameworks, development considerations for iOS, Objective-C, mobile development IDEs, Android development, online emulators and other totally random mobile development things which should have been way out of the scope of this book!

Chapter 7 explores configuration and management, the important recipes from this chapter cover the use of VisualVM to remotely interact with and monitor Tomcat and GlassFish servers using JMX.

There is a recipe in this chapter on the use of JRebel for rapid redeployment. JRebel is a non-free, closed source product, although a free 14-day trial download is available. I try not to use non-free, closed source software, and could not promote the use of this product on my blog.

With Chapter 8 the book finishes with some useful tutorials for using profiling tools to monitor JVM memory usage and utilities to monitor HTTP activity. It covers the use of VisualVM and the jstatd plugin, monitoring TCP connections with Netstat, proxying connection with TCPMon to observe HTTP traffic, and server monitoring with Munin.

Unfortunately the chapter finishes with another recipe for a closed source, non-free product, HTTP Debugger. Again a 14 day trial download is available, but as the software is not FOSS I did not use it.

Overall I think there are some good tips in this book, but some chapters just feel like filler. If you're not experienced already with JEE development, then this isn't the book to start to learn with. If you are experienced then some chapters will seem a bit basic, and I'm sure some will seem a bit off topic. But all JEE developers will take at least a handful of useful recipes from this book.

This is an OK book but not what I was expecting. I'm not sure if the author is trying to take systems approach to JEE development, or just picked some of his favorite topics on the periphery of JEE development and put them together. I'd make sure you have a flick through it before you buy.


Downloads From Packt

Iterate over first n files in a directory with Java 6 and JNA on Linux

$
0
0

I needed to be able to iterate over the first n number of files in a directory, without bringing them all back at once using File.list(). This is simple enough to do in Java 7 using the new DirectoryStream interface, but I only had access to Java 6 for this project.

Similar functionality can be reproduced using calls to native operating system functions using Java Native Access.

This example works on Linux but could easily be adapted for Windows.

DirectoryStream.java

package org.adrianwalker;

import com.sun.jna.Library;
import com.sun.jna.Native;
import com.sun.jna.Pointer;
import com.sun.jna.Structure;
import java.io.Closeable;
import java.io.File;
import java.io.IOException;
import java.util.Iterator;
import java.util.NoSuchElementException;

public final class DirectoryStream implements Iterable<File>, Closeable {

  private String dirName;
  private POSIX posix;
  private Pointer dirp;

  public DirectoryStream(final String dirName) {
    this.dirName = dirName;
    this.posix = (POSIX) Native.loadLibrary("c", POSIX.class);
  }

  @Override
  public Iterator<File> iterator() {

    return new Iterator<File>() {
      private boolean gotNext = false;
      private Dirent file;

      @Override
      public boolean hasNext() {

        if (!gotNext) {
          getNext();
        }

        return file != null;
      }

      @Override
      public File next() {

        if (!gotNext) {
          getNext();
        }

        if (null == file) {
          throw new NoSuchElementException();
        }

        gotNext = false;

        return new File(dirName, file.getName());
      }

      @Override
      public void remove() {
        throw new UnsupportedOperationException();
      }

      private void getNext() {
        if (null == dirp) {
          dirp = posix.opendir(dirName);
        }

        file = posix.readdir(dirp);

        gotNext = true;
      }
    };
  }

  @Override
  public void close() throws IOException {
    posix.closedir(dirp);
  }

  public interface POSIX extends Library {

    public Pointer opendir(String dirname);

    Dirent readdir(Pointer dirp);

    int closedir(Pointer dirp);
  }

  public static final class Dirent extends Structure {

    public long d_ino;
    public long d_off;
    public short d_reclen;
    public char d_type;
    public char[] d_name = new char[256];

    public String getName() {
      return getPointer().getString(8 + 8 + 2 + 1, false);
    }

    public long getInode() {
      return getPointer().getLong(0);
    }

    public void setUseMemory(Pointer p) {
      useMemory(p);
    }
  }
}

Example usage:-

Main.java

package org.adrianwalker;

import java.io.File;
import java.io.IOException;

public final class Main {

  private static final int MAX_FILES = 10;
  private static final String CURRENT_DIR = ".";
  private static final String PARENT_DIR = "..";

  public static void main(final String[] args) throws IOException {

    String dirName = "/tmp/files";
    DirectoryStream directoryStream = new DirectoryStream(dirName);

    int fileCount = 0;
    for (File file : directoryStream) {

      String name = file.getName();
      if (name.equals(CURRENT_DIR) || name.equals(PARENT_DIR)) {
        continue;
      }      
      
      if (fileCount++ == MAX_FILES) {
        break;
      }

      System.out.println(file);
    }
  }
}

Source Code

Usage

Build the project with 'mvn clean install' and run the Main class.

Email Sky Router Public IP Address with Gmail in Python

$
0
0

As far as I know, Sky Broadband don't offer a static IP address. You're going to want a way of keeping track of your externally available dynamic IP, so you can access your machine from outside of your network.

The python script below scrapes your Sky router's status page for your current IP and emails it your Gmail account.

Remember to change the Gmail username and password variables from '????????' to your Gmail authentication credentials. Run this script periodically as a cron job, and as long as you have Gmail access you'll know your IP.

#!/usr/bin/python

import smtplib
from email.mime.text import MIMEText
import urllib2
import re

# router login page
router_login = "http://192.168.0.1"
# router status url
router_status = "http://192.168.0.1/sky_router_status.html"
# router logout page
router_logout = "http://192.168.0.1/sky_logout.html"
# default sky router username and password
router_username = "admin"
router_password = "sky"
# gmail email address
email = "????????@gmail.com"
# gmail password
email_password = "????????"
# gmail smtp settings
server = "smtp.gmail.com"
port = 587
# email subject
subject = "Server IP"

# login to router
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, router_login, router_username, router_password)
authhandler = urllib2.HTTPBasicAuthHandler(passman)
opener = urllib2.build_opener(authhandler)
urllib2.install_opener(opener)
urllib2.urlopen(router_login)

# get the first ip on the status page
ip = urllib2.urlopen(router_status).read()
ip = re.findall(r"[0-9]+(?:\.[0-9]+){3}", ip)
ip = ip[0]

# logout of router
urllib2.urlopen(router_logout)

# check for last ip address
f = open(".last_ip", "a+")
f.seek(0, 0)
last_ip = f.readline()

# has ip changed
if ip != last_ip:

  # store the new ip
  f.truncate(0)
  f.write(ip)

  session = smtplib.SMTP(server, port)

  session.ehlo()
  session.starttls()
  session.login(email, email_password)

  body = "http://" + ip

  msg = MIMEText(body)
  msg['Subject'] = subject
  msg['From'] = email
  msg['To'] = email

  session.sendmail(email, email, msg.as_string())
  session.quit()

f.close()

Properties design pattern and Prototype-based programming in Java

$
0
0

After reading Steve Yegge's post about the properties design pattern and how it can be used to create a prototype-based object system, I thought I'd have a go at an implementation in Java to understand more.

If you haven't read the above post already, it definitely worth a read (or two), and makes everything below make more sense, give it a go.

First, I wanted to define the minimum API needed to get a working properties object as described in the Overview section. A basic properties interface:

Properties.java

package org.adrianwalker.propertiespattern;

interface Properties<N, V> {

  V get(N name);

  V put(N name, V value);

  boolean has(N name);

  V remove(N name);
}

I'll defer describing the implementation until the end, because as soon as you have the basic functionality done there is a load more you're going to want to add.

Once you've got your Properties implementation storing name/value pairs and inheriting from a prototype Properties object stored against a reserved key name, at that point, your going to want to add some way for your objects to simulate methods to manipulate the property values.

Because Java doesn't have first class functions, a methods functionality needs to be encapsulated using the Strategy pattern.

Strategy.java

package org.adrianwalker.propertiespattern;

interface Strategy<N, V, T, A> {

  T execute(Properties<N, V> properties, A... args);
}

In addition to the Strategy interface, I extended the basic Properties interface with an execute method to call a Strategy property:

ExecutableProperites.java

package org.adrianwalker.propertiespattern;

interface ExecutableProperites<N, V, T, A> extends Properties<N, V> {

  T execute(N name, A... args);
}

You're also probably going to want a way of returning an object's property names, possibly filtered with by a regex, or a globing pattern, or partial string match or some kind object comparison. So another extension to the Properties interface:

FilterableProperties.java

package org.adrianwalker.propertiespattern;

import java.util.Set;

interface FilterableProperties<N, V> extends Properties<N, V> {

  Set<N> filter(N filter);
}

And it would be good to have an interface for cloning prototype-based objects:

CloneableProperties.java

package org.adrianwalker.propertiespattern;

interface CloneableProperties<N, V> extends Properties<N, V>, Cloneable {

  CloneableProperties<N, V> clone();
}

Implementation

With all of that out of the way it's time for an implementation. This implementation stores property name/value pairs in a HashMap, and filters using regular expressions:

HashMapProperties.java

package org.adrianwalker.propertiespattern;

import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
import java.util.regex.Pattern;

public final class HashMapProperties<V, T, A> implements
        Properties<String, V>,
        ExecutableProperites<String, V, T, A>,
        CloneableProperties<String, V>,
        FilterableProperties<String, V> {

  public static final String PROTOTYPE = "prototype";
  private final Map<String, V> properties;

  public HashMapProperties() {

    this.properties = Collections.synchronizedMap(new HashMap());
  }

  private HashMapProperties(final Map<String, V> map) {

    this.properties = Collections.synchronizedMap(new HashMap<String, V>(map));
  }

  @Override
  public V get(final String name) {

    if (properties.containsKey(name)) {

      return properties.get(name);
    }

    if (properties.containsKey(PROTOTYPE)) {

      Properties<String, V> prototype =
              (Properties<String, V>) properties.get(PROTOTYPE);

      return prototype.get(name);
    }

    return null;
  }

  @Override
  public V put(final String name, final V value) {

    if (PROTOTYPE.equals(name) && !(value instanceof Properties)) {

      throw new IllegalArgumentException(
              "prototype must be an instance of Properties");
    }

    return properties.put(name, value);
  }

  @Override
  public boolean has(final String name) {

    if (properties.containsKey(name)) {

      return null != properties.get(name);
    }

    if (properties.containsKey(PROTOTYPE)) {

      Properties<String, V> prototype =
              (Properties<String, V>) properties.get(PROTOTYPE);

      return prototype.has(name);
    }

    return false;
  }

  @Override
  public V remove(final String name) {

    if (properties.containsKey(PROTOTYPE)) {

      Properties<String, V> prototype =
              (Properties<String, V>) properties.get(PROTOTYPE);

      if (prototype.has(name)) {

        properties.put(name, null);

        return prototype.get(name);
      }
    }

    if (properties.containsKey(name)) {

      return properties.remove(name);
    }

    return null;
  }

  @Override
  public HashMapProperties<V, T, A> clone() {

    return new HashMapProperties<V, T, A>(properties);
  }

  @Override
  public T execute(final String name, final A... args) {

    V property = get(name);

    if (property instanceof Strategy) {

      return ((Strategy<String, V, T, A>) property).execute(this, args);
    }

    return null;
  }

  @Override
  public Set<String> filter(final String regex) {

    Pattern pattern = Pattern.compile(regex);

    Set<String> filteredNames = new HashSet<String>();

    Set<String> names = properties.keySet();

    synchronized (properties) {
      for (String name : names) {
        if (!PROTOTYPE.equals(name) && pattern.matcher(name).matches()) {
          filteredNames.add(name);
        }
      }
    }

    if (properties.containsKey(PROTOTYPE)) {
      FilterableProperties<String, V> prototype =
              (FilterableProperties<String, V>) properties.get(PROTOTYPE);
      filteredNames.addAll(prototype.filter(regex));
    }

    return filteredNames;
  }
}

Example Usage

The following example shows how you might use the above implementation. A prototype Properties object is created with a 'greeting' property and a 'getGreeting' Strategy property. These properties are inherited by a new Properties object.

The Properties object is cloned and it's greeting property is overridden.

The cloned Properties object's properties are filtered to return the getter Strategy's property name, then this is executed for the Properties object and it's clone:

Example.java

package org.adrianwalker.propertiespattern;

import java.util.Set;
import static org.adrianwalker.propertiespattern.HashMapProperties.PROTOTYPE;

public final class Example {

  public static void main(final String[] args) {
 
    // create a prototype properties object with a greeting and a 
    // strategy which returns the greeting and an argument
 
    Properties prototype = new HashMapProperties();
    prototype.put("greeting", "Hello");
    prototype.put("getGreeting", new Strategy() {
      @Override
      public String execute(final Properties properties, final Object... args) {

        return String.format("%s %s", properties.get("greeting"), args[0]);
      }
    });
 
    // create a properties object which inherit the greeting and 
    // strategy from the prototype

    HashMapProperties properties = new HashMapProperties();
    properties.put(PROTOTYPE, prototype);

    // clone the properties object and override the greeting property

    HashMapProperties clone = properties.clone();
    clone.put("greeting", "Ahoy");

    // filter for the getter properties, then execute the getter
    // on each properties object with a message argument and
    // print the message

    Set<String> getters = clone.filter("get.*");
    for (String getter : getters) {
      System.out.println(properties.execute(getter, "World"));
      System.out.println(clone.execute(getter, "Sailor"));
    }
  }
}

The example above should print the output:

Hello World
Ahoy Sailor

Finally

I've not implemented half of the functionality mentioned in original post, and not a fraction of a fraction of the possible applications for this pattern. The above code and a set of unit tests are available for download at the bottom of this page.

If you need the level of flexibility to warrant using this pattern, then, to quote the original article "use a programming language better suited for implementing the pattern: ideally, one that supports it from the ground up". So expect some JavaScript related posts in the future.

Source Code

Usage

Compile the project with 'mvn clean install'.

Selected Blog Posts Sources In GitHub

Learning JavaScriptMVC Review

Java Collection Literals

$
0
0

The blog post I've had by far the most feedback on is Java Multiline String. It seems to have found a niche with a few programmers for quickly defining formatted SQL and XML for unit testing.

After reading Steve Yegge's post which covered the lack of Java syntax for defining literal data objects, I thought the multiline string code could be extended to provide this kind of syntax for Java Collections.

The code below, like the multiline string code, uses Javadoc comments, annotations and annotation processors to generate code at compile time to initialise and populate a collection.

Collection fields are annotated with an annotation which specifies which annotation processor to use. The annotation processor uses the fields Javadoc comment, which contains a JSON string, to generate additional code to populate the field.

For example, the annotation used to initialise a List field as an ArrayList:

ArrayList.java

package org.adrianwalker.collectionliterals;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Target(ElementType.FIELD)
@Retention(RetentionPolicy.SOURCE)
public @interface ArrayList {
}

The code generation is handled using an annotation processor for each annotation, with common code being inherited from an abstract class:

AbstractCollectionProcessor.java

package org.adrianwalker.collectionliterals;

import com.sun.source.tree.ClassTree;
import com.sun.source.util.Trees;
import com.sun.tools.javac.code.Flags;
import com.sun.tools.javac.code.Symbol;
import com.sun.tools.javac.model.JavacElements;
import com.sun.tools.javac.processing.JavacProcessingEnvironment;
import com.sun.tools.javac.tree.JCTree;
import com.sun.tools.javac.tree.JCTree.JCExpression;
import com.sun.tools.javac.tree.JCTree.JCMethodInvocation;
import com.sun.tools.javac.tree.TreeMaker;
import com.sun.tools.javac.util.Context;
import com.sun.tools.javac.util.List;
import com.sun.tools.javac.util.ListBuffer;
import com.sun.tools.javac.util.Names;
import java.util.Collection;
import java.util.Set;
import javax.annotation.processing.AbstractProcessor;
import javax.annotation.processing.ProcessingEnvironment;
import javax.annotation.processing.RoundEnvironment;
import javax.lang.model.element.Element;
import javax.lang.model.element.TypeElement;
import org.codehaus.jackson.map.ObjectMapper;

public abstract class AbstractCollectionProcessor<C> extends AbstractProcessor {

  private static final String THIS = "this";
  private JavacElements elementUtils;
  private TreeMaker maker;
  private Trees trees;
  private Names names;

  @Override
  public void init(final ProcessingEnvironment procEnv) {
    super.init(procEnv);
    JavacProcessingEnvironment javacProcessingEnv = (JavacProcessingEnvironment) procEnv;
    Context context = javacProcessingEnv.getContext();
    this.elementUtils = javacProcessingEnv.getElementUtils();
    this.maker = TreeMaker.instance(context);
    this.trees = Trees.instance(procEnv);
    this.names = Names.instance(context);
  }

  public JavacElements getElementUtils() {
    return elementUtils;
  }

  public TreeMaker getMaker() {
    return maker;
  }

  public Trees getTrees() {
    return trees;
  }

  public Names getNames() {
    return names;
  }

  @Override
  public boolean process(final Set<? extends TypeElement> annotations, final RoundEnvironment roundEnv) {

    Set<? extends Element> fields = roundEnv.getElementsAnnotatedWith(getAnnotationClass());

    for (Element field : fields) {
      String docComment = elementUtils.getDocComment(field);

      if (null == docComment) {
        continue;
      }

      processField(field, parseJson(docComment));
    }

    return true;
  }

  private C parseJson(final String json) throws ProcessorException {

    ObjectMapper mapper = new ObjectMapper();

    C collection;
    try {
      collection = (C) mapper.readValue(json, getCollectionClass());
    } catch (final Throwable t) {
      throw new ProcessorException(t);
    }

    return collection;
  }

  private void processField(final Element field, final C collection) {

    JCTree fieldTree = elementUtils.getTree(field);
    JCTree.JCVariableDecl fieldVariable = (JCTree.JCVariableDecl) fieldTree;
    Symbol.VarSymbol fieldSym = fieldVariable.sym;
    boolean isStatic = fieldSym.isStatic();
    JCTree.JCExpression fieldEx;

    if (!isStatic) {
      fieldEx = maker.Ident(names.fromString(THIS));
      fieldEx = maker.Select(fieldEx, fieldSym.name);
    } else {
      fieldEx = maker.QualIdent(fieldSym);
    }

    Symbol.ClassSymbol collectionClass = elementUtils.getTypeElement(getCollectionClassName());
    JCTree.JCExpression collectionEx = maker.QualIdent(collectionClass);

    JCTree.JCNewClass newCollection = maker.NewClass(
            null,
            List.<JCTree.JCExpression>nil(),
            collectionEx,
            List.<JCTree.JCExpression>nil(), null);

    JCTree.JCAssign assign = maker.Assign(fieldEx, newCollection);
    JCTree.JCStatement assignStatement = maker.Exec(assign);

    JCExpression collectionMethodEx = maker.Select(fieldEx, names.fromString(getCollectionMethodName()));

    ListBuffer blockBuffer = new ListBuffer();
    blockBuffer.add(assignStatement);
    blockBuffer.appendList(processCollection(collection, collectionMethodEx));

    long flags = Flags.BLOCK;
    if (isStatic) {
      flags |= Flags.STATIC;
    }

    JCTree.JCBlock block = maker.Block(flags, blockBuffer.toList());

    TypeElement type = (TypeElement) field.getEnclosingElement();
    ClassTree cls = trees.getTree(type);

    JCTree.JCClassDecl cd = (JCTree.JCClassDecl) cls;

    cd.defs = cd.defs.append(block);
  }

  public List processCollection(final C collection, final JCTree.JCExpression collectionMethodEx) {
    ListBuffer collectionMethodBuffer = new ListBuffer();

    for (Object element : (Collection) collection) {

      ListBuffer addBuffer = new ListBuffer();
      addBuffer.add(getMaker().Literal(element));

      JCMethodInvocation addInv = getMaker().Apply(
              List.<JCTree.JCExpression>nil(),
              collectionMethodEx,
              addBuffer.toList());
      JCTree.JCStatement addStatement = getMaker().Exec(addInv);

      collectionMethodBuffer.add(addStatement);
    }

    return collectionMethodBuffer.toList();
  }

  public abstract Class getAnnotationClass();

  public abstract String getCollectionClassName();

  public abstract Class getCollectionClass();

  public abstract String getCollectionMethodName();
}

This class can be extended to create a concrete implementation, for an ArrayList field:

ArrayListProcessor.java

package org.adrianwalker.collectionliterals;

import java.util.ArrayList;
import javax.annotation.processing.SupportedAnnotationTypes;
import javax.annotation.processing.SupportedSourceVersion;
import javax.lang.model.SourceVersion;

@SupportedAnnotationTypes({"org.adrianwalker.collectionliterals.ArrayList"})
@SupportedSourceVersion(SourceVersion.RELEASE_6)
public final class ArrayListProcessor extends AbstractCollectionProcessor<ArrayList<Object>> {

  @Override
  public Class getAnnotationClass() {
    return org.adrianwalker.collectionliterals.ArrayList.class;
  }

  @Override
  public Class getCollectionClass() {
    return java.util.ArrayList.class;
  }

  @Override
  public String getCollectionClassName() {
    return getCollectionClass().getName();
  }

  @Override
  public String getCollectionMethodName() {
    return "add";
  }
}

Because Java Maps don't implement the Collection interface, the processor for a HashMap field is a bit more complicated:

HashMapProcessor.java

package org.adrianwalker.collectionliterals;

import com.sun.tools.javac.tree.JCTree;
import com.sun.tools.javac.tree.JCTree.JCMethodInvocation;
import com.sun.tools.javac.util.List;
import com.sun.tools.javac.util.ListBuffer;
import java.util.HashMap;
import java.util.Map.Entry;
import javax.annotation.processing.SupportedAnnotationTypes;
import javax.annotation.processing.SupportedSourceVersion;
import javax.lang.model.SourceVersion;

@SupportedAnnotationTypes({"org.adrianwalker.collectionliterals.HashMap"})
@SupportedSourceVersion(SourceVersion.RELEASE_6)
public final class HashMapProcessor extends AbstractCollectionProcessor<HashMap<String, Object>> {

  @Override
  public List processCollection(final HashMap<String, Object> collection, final JCTree.JCExpression collectionMethodEx) {
    ListBuffer collectionMethodBuffer = new ListBuffer();

    for (Entry<String, Object> entry : collection.entrySet()) {

      ListBuffer putBuffer = new ListBuffer();
      putBuffer.add(getMaker().Literal(entry.getKey()));
      putBuffer.add(getMaker().Literal(entry.getValue()));

      JCMethodInvocation putInv = getMaker().Apply(
              List.<JCTree.JCExpression>nil(),
              collectionMethodEx,
              putBuffer.toList());
      JCTree.JCStatement putStatement = getMaker().Exec(putInv);

      collectionMethodBuffer.add(putStatement);
    }

    return collectionMethodBuffer.toList();
  }

  @Override
  public Class getAnnotationClass() {
    return org.adrianwalker.collectionliterals.HashMap.class;
  }

  @Override
  public Class getCollectionClass() {
    return java.util.HashMap.class;
  }

  @Override
  public String getCollectionClassName() {
    return getCollectionClass().getName();
  }

  @Override
  public String getCollectionMethodName() {
    return "put";
  }
}

The code to use the annotations would look something like this for a class with static fields:

CollectionLiteralUsageStatic.java

package org.adrianwalker.collectionliterals;

import java.util.List;
import java.util.Map;
import java.util.Set;

public final class CollectionLiteralUsageStatic {

  /**
   [1,2,3,4,5,6,7,8,9,10]
   */
  @ArrayList
  private static List<Integer> list;
  /**
   ["0","1","2","3","4","5","6","7",
   "8","9","A","B","C","D","E","F"]
   */
  @HashSet
  private static Set<String> set;  
  /**
   {
     "name":"Adrian Walker", 
     "age":31 
   }
   */
  @HashMap
  private static Map<String, Object> map;

  public static void main(final String[] args) {
    System.out.println(list);
    System.out.println(set);
    System.out.println(map);
  }
}

And example code for classes with instance fields:

package org.adrianwalker.collectionliterals;

import java.util.List;
import java.util.Map;
import java.util.Set;

public final class CollectionLiteralUsage {

  /**
   [1,2,3,4,5,6,7,8,9,10]
   */
  @LinkedList
  private List<Integer> list;
  /**
   ["0","1","2","3","4","5","6","7", 
   "8","9","A","B","C","D","E","F"]
   */
  @HashSet
  private Set<String> set;
  /**
   {
     "name":"Adrian Walker", 
     "age":31 
   }
   */
  @HashMap
  private Map<String, Object> map;

  public List getList() {
    return list;
  }

  public Set<String> getSet() {
    return set;
  }

  public Map getMap() {
    return map;
  }

  public static void main(final String[] args) {
    CollectionLiteralUsage collectionUsage = new CollectionLiteralUsage();
    System.out.println(collectionUsage.getList());
    System.out.println(collectionUsage.getSet());
    System.out.println(collectionUsage.getMap());
  }
}

As before with the multiline code, remember to include an <annotationProcessor> element when compiling classes which use the annotations when using Maven.

pom.xml

...
<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-compiler-plugin</artifactId><version>2.3.2</version><configuration><source>1.6</source><target>1.6</target><annotationProcessors><annotationProcessor>
        org.adrianwalker.collectionliterals.ArrayListProcessor
      </annotationProcessor><annotationProcessor>
        org.adrianwalker.collectionliterals.LinkedListProcessor
      </annotationProcessor><annotationProcessor>
        org.adrianwalker.collectionliterals.HashSetProcessor
      </annotationProcessor><annotationProcessor>
        org.adrianwalker.collectionliterals.HashMapProcessor
      </annotationProcessor></annotationProcessors></configuration></plugin>
...

A commenter on Stack Overflow had expressed an issue with this approach having 'a hard dependency on Maven', which is not right. The javac command has direct support for specifying annotation processors: http://docs.oracle.com/javase/6/docs/technotes/tools/solaris/javac.html#processing

Limitations

Currently the code does not support nested data structures, for example this JSON string cannot be used to generate code:

{
  "name":"Adrian Walker",
  "age":31,
  "address":[
    "line1",
    "line2",
    "line3"
  ]
}

Source Code

  • Annotations, annotation processors and example usage code available in GitHub - collection-literals

Rigging the National Lottery (Is Hard)

$
0
0

As I’ve never won the National Lottery jackpot, I’m pretty sure the whole thing is rigged.

If you were in charge of taking peoples hard earned cash off of them and redistributing it back to a few of them, I bet you’d try and minimize the pay-out and pocket as much as you could. Now I’m not saying Camelot UK Lotteries Limited actually does this and The National Lottery is rigged, but you know, they definitely do and it is.

On draw nights, you can’t buy tickets between 7.30pm and 9.00pm before the winning balls are selected. Have you ever wondered what goes on for that hour and a half? I’ll tell you what; they are calculating which balls to select to give you the least money possible.

To minimize the pay-out you would need to compare all the possible combinations of six main balls plus the bonus ball, with all the entry lines which players have picked. Each comparison would check to see how many entry numbers matched the draw numbers, and the number of matches counted for all the entry lines. Finally, for each of the draw ball combinations, the total prize cost can be calculated using the stored matches count and the prize value for that number of matches.

Looking at the lotto prize breakdown from a past week (http://www.national-lottery.co.uk/player/lotto/results/prizeBreakdown.ftl):

MatchesNo. of winners£s per winnerPrize fund
Match 61£7,959,312£7,959,312
Match 5 plus Bonus7£68,525£479,675
Match 5176£2,314£407,264
Match 412,448£173£2,153,504
Match 3256,646£25£6,416,150
Lotto Raffle100£20,000£2,000,000
Totals269,378£19,415,905

The total prize fund was about £20,000,000. From the Lotto website (http://www.national-lottery.co.uk/player/p/goodcausesandwinners/wherethemoneygoes.ftl), 50% of the money spent goes to the prize fund, which means the total money spent buying tickets was about £40,000,000, which at £2 per ticket gives us:

£40,000,000 / £2 = 20,000,000 entries.

Even if we could compare one entry line to all combinations of draw numbers every second, this would still take:

20,000,000 seconds / (60 * 60 * 24) ~ 232 days

Way too long to process in 90 minutes.

To rig the lotto draw, we're going to need some serious speed. The following is an unoptimised implementation in Java to process ten randomly generated entry lines, to use as a base line benchmark:

Lotto1.java

package org.adrianwalker.lotto;

import java.util.Random;

public final class Lotto1 {

  /*
   * Estimated lotto prizes:
   *   https://www.national-lottery.co.uk/player/p/help/aboutlotto/prizecalculation.ftl
   */
  private static final int MATCH_6_PRIZE = 5000000;
  private static final int MATCH_5_PLUS_BONUS_PRIZE = 50000;
  private static final int MATCH_5_PRIZE = 1000;
  private static final int MATCH_4_PRIZE = 100;
  private static final int MATCH_3_PRIZE = 25;
  // match types
  private static final int MATCH_5_PLUS_BONUS = 7;
  private static final int MATCH_6 = 6;
  private static final int MATCH_5 = 5;
  private static final int MATCH_4 = 4;
  private static final int MATCH_3 = 3;
  // define lowest and highest ball numbers
  private static final int LOW_BALL = 1;
  private static final int HIGH_BALL = 49;
  // array sizes
  private static final int MAIN_BALLS_SIZE = 6;
  private static final int ENTRY_BALLS_SIZE = 6;
  private static final int ENTRIES_SIZE = 10;

  public static void main(final String[] args) {

    // lotto entries, could be read from a database or file?
    int[][] entries = new int[ENTRIES_SIZE][ENTRY_BALLS_SIZE];

    // draw balls for random entries
    int[] balls = {
      1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
      11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
      21, 22, 23, 24, 25, 26, 27, 28, 29, 30,
      31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
      41, 42, 43, 44, 45, 46, 47, 48, 49
    };

    // create random entries for testing
    for (int i = 0; i < ENTRIES_SIZE; i++) {
      shuffle(balls);
      entries[i][0] = balls[0];
      entries[i][1] = balls[1];
      entries[i][2] = balls[2];
      entries[i][3] = balls[3];
      entries[i][4] = balls[4];
      entries[i][5] = balls[5];
    }

    long start = System.currentTimeMillis();

    // minimum values
    long minCost = Long.MAX_VALUE;
    int[] minCostMainBalls = new int[MAIN_BALLS_SIZE];
    int minCostBonusBall = -1;

    // maximum values
    long maxCost = Long.MIN_VALUE;
    int[] maxCostMainBalls = new int[MAIN_BALLS_SIZE];
    int maxCostBonusBall = -1;

    // iterate over main ball combinations
    int[] mainBalls = new int[MAIN_BALLS_SIZE];
    for (mainBalls[0] = LOW_BALL; mainBalls[0] <= HIGH_BALL; mainBalls[0]++) {
      for (mainBalls[1] = mainBalls[0] + 1; mainBalls[1] <= HIGH_BALL; mainBalls[1]++) {
        for (mainBalls[2] = mainBalls[1] + 1; mainBalls[2] <= HIGH_BALL; mainBalls[2]++) {
          for (mainBalls[3] = mainBalls[2] + 1; mainBalls[3] <= HIGH_BALL; mainBalls[3]++) {
            for (mainBalls[4] = mainBalls[3] + 1; mainBalls[4] <= HIGH_BALL; mainBalls[4]++) {
              for (mainBalls[5] = mainBalls[4] + 1; mainBalls[5] <= HIGH_BALL; mainBalls[5]++) {

                // iterate over bonus balls
                for (int bonusBall = LOW_BALL; bonusBall <= HIGH_BALL; bonusBall++) {

                  // skip bonus ball if its in the main balls
                  if (contains(mainBalls, bonusBall)) {
                    continue;
                  }

                  // init match counts to zero
                  int[] matchesCount = {0, 0, 0, 0, 0, 0, 0, 0};

                  // iterate over entries
                  for (int[] entry : entries) {
                    // count matches
                    int matches = matches(entry, mainBalls, bonusBall);
                    // increment matches for number of balls matched
                    matchesCount[matches]++;
                  }

                  // get total cost of draw
                  long cost = cost(matchesCount);

                  // keep track of highest/lowest draw costs
                  if (cost < minCost) {
                    minCost = cost;
                    minCostMainBalls = mainBalls.clone();
                    minCostBonusBall = bonusBall;
                  } else if (cost > maxCost) {
                    maxCost = cost;
                    maxCostMainBalls = mainBalls.clone();
                    maxCostBonusBall = bonusBall;
                  }
                }
              }
            }
          }
        }
      }
    }

    long end = System.currentTimeMillis();

    print(start, end,
            minCost, minCostMainBalls, minCostBonusBall,
            maxCost, maxCostMainBalls, maxCostBonusBall);
  }

  private static boolean contains(final int[] a, final int key) {

    for (int x : a) {
      if (x == key) {
        return true;
      }
    }

    return false;
  }

  private static void shuffle(final int[] a) {

    Random rand = new Random(System.currentTimeMillis());

    for (int i = 0; i < a.length; i++) {

      int index = rand.nextInt(i + 1);
      int x = a[index];
      a[index] = a[i];
      a[i] = x;
    }
  }

  private static int matches(final int[] entry, final int[] mainBalls, final int bonusBall) {

    int matches = 0;
    boolean bonusMatch = false;

    for (int entryNumber : entry) {
      for (int mainBall : mainBalls) {

        if (mainBall == entryNumber) {
          matches++;
        } else if (entryNumber == bonusBall) {
          bonusMatch = true;
        }
      }
    }

    if (matches == MATCH_5 && bonusMatch) {
      matches = MATCH_5_PLUS_BONUS;
    }

    return matches;
  }

  private static long cost(final int[] matchesCount) {

    long cost = 0;

    // add jackpot prize
    if (matchesCount[MATCH_6] > 0) {
      cost = MATCH_6_PRIZE;
    }

    // add lesser prizes, multiplied by number of winners
    cost += matchesCount[MATCH_5_PLUS_BONUS] * MATCH_5_PLUS_BONUS_PRIZE
            + matchesCount[MATCH_5] * MATCH_5_PRIZE
            + matchesCount[MATCH_4] * MATCH_4_PRIZE
            + matchesCount[MATCH_3] * MATCH_3_PRIZE;

    return cost;
  }

  private static void print(
          final long start, final long end,
          final long minCost, final int[] minCostMainBalls, final int minCostBonusBall,
          final long maxCost, final int[] maxCostMainBalls, final int maxCostBonusBall) {

    System.out.printf("\n--- entries ---\n");
    System.out.printf("random entries processed = %s\n", ENTRIES_SIZE);

    System.out.printf("\n--- time ---\n");
    System.out.printf("elapsed time = %s seconds\n", (end - start) / 1000);

    System.out.printf("\n--- min ---\n");
    System.out.printf("minimum cost = %s\n", minCost);
    System.out.printf("minimum main balls = %s,%s,%s,%s,%s,%s\n",
            minCostMainBalls[0],
            minCostMainBalls[1],
            minCostMainBalls[2],
            minCostMainBalls[3],
            minCostMainBalls[4],
            minCostMainBalls[5]);
    System.out.printf("minimum bonus ball = %s\n",
            minCostBonusBall == -1 ? "any" : minCostBonusBall);

    System.out.printf("\n--- max ---\n");
    System.out.printf("maximum cost = %s\n", maxCost);
    System.out.printf("maximum main balls = %s,%s,%s,%s,%s,%s\n",
            maxCostMainBalls[0],
            maxCostMainBalls[1],
            maxCostMainBalls[2],
            maxCostMainBalls[3],
            maxCostMainBalls[4],
            maxCostMainBalls[5]);
    System.out.printf("maximum bonus ball = %s\n",
            maxCostBonusBall == -1 ? "any" : maxCostBonusBall);
  }
}

On my very old L7500 1.60GHz laptop the output looks like this:

--- entries ---
random entries processed = 10

--- time ---
elapsed time = 636 seconds

--- min ---
minimum cost = 0
minimum main balls = 1,2,3,4,5,6
minimum bonus ball = 7

--- max ---
maximum cost = 5000000
maximum main balls = 1,6,8,15,22,37
maximum bonus ball = 2

636 seconds for ten entries! Way too slow! There are some optimisations we can perform to make this faster:

  1. We only care about the bonus ball if we have 5 matching main balls - this loop can be executed only when we need it.
  2. An intersection of arrays can be used to count the main ball array/entry array matches.
  3. Sorted entry arrays can be used with a binary search to find bonus ball matches.

Lotto2.java

package org.adrianwalker.lotto;

import java.util.Arrays;
import java.util.Random;

public final class Lotto2 {

  /*
   * Estimated lotto prizes:
   *   https://www.national-lottery.co.uk/player/p/help/aboutlotto/prizecalculation.ftl
   */
  private static final int MATCH_6_PRIZE = 5000000;
  private static final int MATCH_5_PLUS_BONUS_PRIZE = 50000;
  private static final int MATCH_5_PRIZE = 1000;
  private static final int MATCH_4_PRIZE = 100;
  private static final int MATCH_3_PRIZE = 25;
  // match types
  private static final int MATCH_5_PLUS_BONUS = 7;
  private static final int MATCH_6 = 6;
  private static final int MATCH_5 = 5;
  private static final int MATCH_4 = 4;
  private static final int MATCH_3 = 3;
  // define lowest and highest ball numbers
  private static final int LOW_BALL = 1;
  private static final int HIGH_BALL = 49;
  // array sizes
  private static final int MAIN_BALLS_SIZE = 6;
  private static final int ENTRY_BALLS_SIZE = 6;
  private static final int ENTRIES_SIZE = 10;

  public static void main(final String[] args) {

    // lotto entries, could be read from a database or file?
    int[][] entries = new int[ENTRIES_SIZE][ENTRY_BALLS_SIZE];

    // draw balls for random entries
    int[] balls = {
      1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
      11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
      21, 22, 23, 24, 25, 26, 27, 28, 29, 30,
      31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
      41, 42, 43, 44, 45, 46, 47, 48, 49
    };

    // create random entries for testing
    for (int i = 0; i < ENTRIES_SIZE; i++) {
      shuffle(balls);
      entries[i][0] = balls[0];
      entries[i][1] = balls[1];
      entries[i][2] = balls[2];
      entries[i][3] = balls[3];
      entries[i][4] = balls[4];
      entries[i][5] = balls[5];
      Arrays.sort(entries[i]);
    }

    long start = System.currentTimeMillis();

    // minimum values
    long minCost = Long.MAX_VALUE;
    int[] minCostMainBalls = new int[MAIN_BALLS_SIZE];
    int minCostBonusBall = -1;

    // maximum values
    long maxCost = Long.MIN_VALUE;
    int[] maxCostMainBalls = new int[MAIN_BALLS_SIZE];
    int maxCostBonusBall = -1;

    // iterate over main ball combinations
    int[] mainBalls = new int[MAIN_BALLS_SIZE];
    for (mainBalls[0] = LOW_BALL; mainBalls[0] <= HIGH_BALL; mainBalls[0]++) {
      for (mainBalls[1] = mainBalls[0] + 1; mainBalls[1] <= HIGH_BALL; mainBalls[1]++) {
        for (mainBalls[2] = mainBalls[1] + 1; mainBalls[2] <= HIGH_BALL; mainBalls[2]++) {
          for (mainBalls[3] = mainBalls[2] + 1; mainBalls[3] <= HIGH_BALL; mainBalls[3]++) {
            for (mainBalls[4] = mainBalls[3] + 1; mainBalls[4] <= HIGH_BALL; mainBalls[4]++) {
              for (mainBalls[5] = mainBalls[4] + 1; mainBalls[5] <= HIGH_BALL; mainBalls[5]++) {

                // init match counts to zero
                int[] matchesCount = {0, 0, 0, 0, 0, 0, 0, 0};

                // init bonus ball
                int bonusBall = -1;

                // iterate over entries
                for (int[] entry : entries) {

                  // count matches
                  int matches = intersection(entry, mainBalls);

                  // if 5 matches check the bonus ball
                  if (matches == MATCH_5) {

                    // iterate over bonus balls
                    for (bonusBall = LOW_BALL; bonusBall <= HIGH_BALL; bonusBall++) {

                      // skip bonus ball if its in the main balls
                      if (contains(mainBalls, bonusBall)) {
                        continue;
                      }

                      // break if the entries contain the bonus ball
                      if (contains(entry, bonusBall)) {
                        matches = MATCH_5_PLUS_BONUS;
                        break;
                      }
                    }
                  }

                  // increment matches for number of balls matched
                  matchesCount[matches]++;
                }

                // get total cost of draw
                long cost = cost(matchesCount);

                // keep track of highest/lowest draw costs
                if (cost < minCost) {
                  minCost = cost;
                  minCostMainBalls = mainBalls.clone();
                  minCostBonusBall = bonusBall;
                } else if (cost > maxCost) {
                  maxCost = cost;
                  maxCostMainBalls = mainBalls.clone();
                  maxCostBonusBall = bonusBall;
                }
              }
            }
          }
        }
      }
    }

    long end = System.currentTimeMillis();

    print(start, end,
            minCost, minCostMainBalls, minCostBonusBall,
            maxCost, maxCostMainBalls, maxCostBonusBall);
  }

  private static boolean contains(final int[] a, final int key) {

    return Arrays.binarySearch(a, key) != -1;
  }

  private static void shuffle(final int[] a) {

    Random rand = new Random(System.currentTimeMillis());

    for (int i = 0; i < a.length; i++) {

      int index = rand.nextInt(i + 1);
      int x = a[index];
      a[index] = a[i];
      a[i] = x;
    }
  }

  private static int intersection(final int[] a1, final int[] a2) {

    int count = 0;
    int i = 0;
    int j = 0;
    int a1length = a1.length;
    int a2length = a2.length;

    while (i < a1length && j < a2length) {
      if (a1[i] == a2[j]) {
        count++;
        i++;
        j++;
      } else if (a1[i] < a2[j]) {
        i++;
      } else if (a1[i] > a2[j]) {
        j++;
      }
    }

    return count;
  }

  private static long cost(final int[] matchesCount) {

    long cost = 0;

    // add jackpot prize
    if (matchesCount[MATCH_6] > 0) {
      cost = MATCH_6_PRIZE;
    }

    // add lesser prizes, multiplied by number of winners
    cost += matchesCount[MATCH_5_PLUS_BONUS] * MATCH_5_PLUS_BONUS_PRIZE
            + matchesCount[MATCH_5] * MATCH_5_PRIZE
            + matchesCount[MATCH_4] * MATCH_4_PRIZE
            + matchesCount[MATCH_3] * MATCH_3_PRIZE;

    return cost;
  }

  private static void print(
          final long start, final long end,
          final long minCost, final int[] minCostMainBalls, final int minCostBonusBall,
          final long maxCost, final int[] maxCostMainBalls, final int maxCostBonusBall) {

    System.out.printf("\n--- entries ---\n");
    System.out.printf("random entries processed = %s\n", ENTRIES_SIZE);

    System.out.printf("\n--- time ---\n");
    System.out.printf("elapsed time = %s seconds\n", (end - start) / 1000);

    System.out.printf("\n--- min ---\n");
    System.out.printf("minimum cost = %s\n", minCost);
    System.out.printf("minimum main balls = %s,%s,%s,%s,%s,%s\n",
            minCostMainBalls[0],
            minCostMainBalls[1],
            minCostMainBalls[2],
            minCostMainBalls[3],
            minCostMainBalls[4],
            minCostMainBalls[5]);
    System.out.printf("minimum bonus ball = %s\n",
            minCostBonusBall == -1 ? "any" : minCostBonusBall);

    System.out.printf("\n--- max ---\n");
    System.out.printf("maximum cost = %s\n", maxCost);
    System.out.printf("maximum main balls = %s,%s,%s,%s,%s,%s\n",
            maxCostMainBalls[0],
            maxCostMainBalls[1],
            maxCostMainBalls[2],
            maxCostMainBalls[3],
            maxCostMainBalls[4],
            maxCostMainBalls[5]);
    System.out.printf("maximum bonus ball = %s\n",
            maxCostBonusBall == -1 ? "any" : maxCostBonusBall);
  }
}

The output from this optimised version looks like this:

--- entries ---
random entries processed = 10

--- time ---
elapsed time = 17 seconds

--- min ---
minimum cost = 0
minimum main balls = 1,2,3,4,5,6
minimum bonus ball = any

--- max ---
maximum cost = 5000025
maximum main balls = 4,7,31,34,36,43
maximum bonus ball = any

17 seconds is a massive performance improvement over the previous code, but still not the sub second per entry speed we require.

The following code uses a long integer to represent the main balls as a bit mask, each 1 bit's position representing a main ball number. This bit mask can be used as a lookup to check entry numbers against:

Lotto3.java

package org.adrianwalker.lotto;

import java.util.Random;

public final class Lotto3 {

  /*
   * Estimated lotto prizes:
   *   https://www.national-lottery.co.uk/player/p/help/aboutlotto/prizecalculation.ftl
   */
  private static final int MATCH_6_PRIZE = 5000000;
  private static final int MATCH_5_PLUS_BONUS_PRIZE = 50000;
  private static final int MATCH_5_PRIZE = 1000;
  private static final int MATCH_4_PRIZE = 100;
  private static final int MATCH_3_PRIZE = 25;
  // match types
  private static final int MATCH_5_PLUS_BONUS = 7;
  private static final int MATCH_6 = 6;
  private static final int MATCH_5 = 5;
  private static final int MATCH_4 = 4;
  private static final int MATCH_3 = 3;
  // define lowest and highest ball numbers
  private static final int LOW_BALL = 1;
  private static final int HIGH_BALL = 49;
  // array sizes
  private static final int MAIN_BALLS_SIZE = 6;
  private static final int ENTRY_BALLS_SIZE = 6;
  private static final int ENTRIES_SIZE = 10;

  public static void main(final String[] args) {

    // lotto entries, could be read from a database or file?
    int[][] entries = new int[ENTRIES_SIZE][ENTRY_BALLS_SIZE];

    // draw balls for random entries
    int[] balls = {
      1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
      11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
      21, 22, 23, 24, 25, 26, 27, 28, 29, 30,
      31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
      41, 42, 43, 44, 45, 46, 47, 48, 49
    };

    // create random entries for testing
    for (int i = 0; i < ENTRIES_SIZE; i++) {
      shuffle(balls);
      entries[i][0] = balls[0];
      entries[i][1] = balls[1];
      entries[i][2] = balls[2];
      entries[i][3] = balls[3];
      entries[i][4] = balls[4];
      entries[i][5] = balls[5];
    }

    long start = System.currentTimeMillis();

    // minimum values
    long minCost = Long.MAX_VALUE;
    int[] minCostMainBalls = new int[MAIN_BALLS_SIZE];
    int minCostBonusBall = -1;

    // maximum values
    long maxCost = Long.MIN_VALUE;
    int[] maxCostMainBalls = new int[MAIN_BALLS_SIZE];
    int maxCostBonusBall = -1;

    // iterate over main ball combinations
    int[] mainBalls = new int[MAIN_BALLS_SIZE];
    for (mainBalls[0] = LOW_BALL; mainBalls[0] <= HIGH_BALL; mainBalls[0]++) {
      for (mainBalls[1] = mainBalls[0] + 1; mainBalls[1] <= HIGH_BALL; mainBalls[1]++) {
        for (mainBalls[2] = mainBalls[1] + 1; mainBalls[2] <= HIGH_BALL; mainBalls[2]++) {
          for (mainBalls[3] = mainBalls[2] + 1; mainBalls[3] <= HIGH_BALL; mainBalls[3]++) {
            for (mainBalls[4] = mainBalls[3] + 1; mainBalls[4] <= HIGH_BALL; mainBalls[4]++) {
              for (mainBalls[5] = mainBalls[4] + 1; mainBalls[5] <= HIGH_BALL; mainBalls[5]++) {

                // create bitmask for match lookup
                long mainBallsBitmask = bitmask(mainBalls);

                // init match counts to zero
                int[] matchesCount = {0, 0, 0, 0, 0, 0, 0, 0};

                // init bonus ball
                int bonusBall = -1;

                // iterate over entries
                for (int[] entry : entries) {

                  // count matches
                  int matches = matches(entry, mainBallsBitmask);

                  // if 5 matches check the bonus ball
                  if (matches == MATCH_5) {

                    // iterate over bonus balls
                    for (bonusBall = LOW_BALL; bonusBall <= HIGH_BALL; bonusBall++) {

                      // skip bonus ball if its in the main balls
                      if (bitSet(mainBallsBitmask, bonusBall)) {
                        continue;
                      }

                      long entryBitmask = bitmask(entry);

                      // break if the entries contain the bonus ball
                      if (bitSet(entryBitmask, bonusBall)) {
                        matches = MATCH_5_PLUS_BONUS;
                        break;
                      }
                    }
                  }

                  // increment matches for number of balls matched
                  matchesCount[matches]++;
                }

                // get total cost of draw
                long cost = cost(matchesCount);

                // keep track of highest/lowest draw costs
                if (cost < minCost) {
                  minCost = cost;
                  minCostMainBalls = mainBalls.clone();
                  minCostBonusBall = bonusBall;
                } else if (cost > maxCost) {
                  maxCost = cost;
                  maxCostMainBalls = mainBalls.clone();
                  maxCostBonusBall = bonusBall;
                }
              }
            }
          }
        }
      }
    }

    long end = System.currentTimeMillis();

    print(start, end,
            minCost, minCostMainBalls, minCostBonusBall,
            maxCost, maxCostMainBalls, maxCostBonusBall);
  }

  private static void shuffle(final int[] a) {

    Random rand = new Random(System.currentTimeMillis());

    for (int i = 0; i < a.length; i++) {

      int index = rand.nextInt(i + 1);
      int x = a[index];
      a[index] = a[i];
      a[i] = x;
    }
  }

  private static boolean bitSet(final long bitMask, final int i) {
    return (bitMask & (1L << i)) > 0;
  }

  private static long setBit(final long bitmask, final int i) {
    return bitmask | (1L << i);
  }

  private static long bitmask(final int[] a) {

    long bitmask = 0;

    for (int i : a) {
      bitmask = setBit(bitmask, i);
    }

    return bitmask;
  }

  private static int matches(final int[] entry, final long mainBallsBitmask) {

    int matches = 0;

    for (int number : entry) {

      if (bitSet(mainBallsBitmask, number)) {
        matches++;
      }
    }

    return matches;
  }

  private static long cost(final int[] matchesCount) {

    long cost = 0;

    // add jackpot prize
    if (matchesCount[MATCH_6] > 0) {
      cost = MATCH_6_PRIZE;
    }

    // add lesser prizes, multiplied by number of winners
    cost += matchesCount[MATCH_5_PLUS_BONUS] * MATCH_5_PLUS_BONUS_PRIZE
            + matchesCount[MATCH_5] * MATCH_5_PRIZE
            + matchesCount[MATCH_4] * MATCH_4_PRIZE
            + matchesCount[MATCH_3] * MATCH_3_PRIZE;

    return cost;
  }

  private static void print(
          final long start, final long end,
          final long minCost, final int[] minCostMainBalls, final int minCostBonusBall,
          final long maxCost, final int[] maxCostMainBalls, final int maxCostBonusBall) {

    System.out.printf("\n--- entries ---\n");
    System.out.printf("random entries processed = %s\n", ENTRIES_SIZE);

    System.out.printf("\n--- time ---\n");
    System.out.printf("elapsed time = %s seconds\n", (end - start) / 1000);

    System.out.printf("\n--- min ---\n");
    System.out.printf("minimum cost = %s\n", minCost);
    System.out.printf("minimum main balls = %s,%s,%s,%s,%s,%s\n",
            minCostMainBalls[0],
            minCostMainBalls[1],
            minCostMainBalls[2],
            minCostMainBalls[3],
            minCostMainBalls[4],
            minCostMainBalls[5]);
    System.out.printf("minimum bonus ball = %s\n",
            minCostBonusBall == -1 ? "any" : minCostBonusBall);

    System.out.printf("\n--- max ---\n");
    System.out.printf("maximum cost = %s\n", maxCost);
    System.out.printf("maximum main balls = %s,%s,%s,%s,%s,%s\n",
            maxCostMainBalls[0],
            maxCostMainBalls[1],
            maxCostMainBalls[2],
            maxCostMainBalls[3],
            maxCostMainBalls[4],
            maxCostMainBalls[5]);
    System.out.printf("maximum bonus ball = %s\n",
            maxCostBonusBall == -1 ? "any" : maxCostBonusBall);
  }
}

The output from the bit mask optimised version looks like this:

--- entries ---
random entries processed = 10

--- time ---
elapsed time = 4 seconds

--- min ---
minimum cost = 0
minimum main balls = 1,2,4,5,6,8
minimum bonus ball = any

--- max ---
maximum cost = 5000025
maximum main balls = 2,20,27,43,46,47
maximum bonus ball = any

Four seconds for ten entries is approaching the sub second per entry performance we would need.

Obviously a faster algorithm is preferable to throwing more CPU at a problem, but porting the above Java code to C allows us to more effectively use the CPU we have, and gives us some speed improvement for free:

lotto.c

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <limits.h>
#include <time.h>

#define FALSE 0
#define TRUE !FALSE
#define SIZEOF(a) (sizeof a / sizeof a[0])

/*
 * Estimated lotto prizes:
 *   https://www.national-lottery.co.uk/player/p/help/aboutlotto/prizecalculation.ftl
 */
#define MATCH_6_PRIZE 5000000
#define MATCH_5_PLUS_BONUS_PRIZE 50000
#define MATCH_5_PRIZE 1000
#define MATCH_4_PRIZE 100
#define MATCH_3_PRIZE 25
// match types
#define MATCH_5_PLUS_BONUS 7
#define MATCH_6 6
#define MATCH_5 5
#define MATCH_4 4
#define MATCH_3 3
// define lowest and highest ball numbers
#define LOW_BALL 1
#define HIGH_BALL 49
// array sizes
#define MAIN_BALLS_SIZE 6
#define ENTRY_BALLS_SIZE 6
#define MATCHES_COUNT_SIZE 8
#define ENTRIES_SIZE 10

void shuffle(int a[], int size);
int bitSet(long bitMask, int i);
long setBit(long bitmask, int i);
long bitmask(int a[], int size);
int matches(int entry[ENTRY_BALLS_SIZE], long mainBallsBitmask);
long cost(int matchesCount[MATCHES_COUNT_SIZE]);
void print(
        time_t start, time_t end,
        long minCost, int minCostMainBalls[MAIN_BALLS_SIZE], int minCostBonusBall,
        long maxCost, int maxCostMainBalls[MAIN_BALLS_SIZE], int maxCostBonusBall);

int main(int argc, char** argv) {

  // lotto entries, could be read from a database or file?
  int entries[ENTRIES_SIZE][ENTRY_BALLS_SIZE];

  // draw balls for random entries
  int balls[] = {
    1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
    11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
    21, 22, 23, 24, 25, 26, 27, 28, 29, 30,
    31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
    41, 42, 43, 44, 45, 46, 47, 48, 49
  };

  // create random entries for testing
  int i;
  for (i = 0; i < ENTRIES_SIZE; i++) {
    shuffle(balls, SIZEOF(balls));
    entries[i][0] = balls[0];
    entries[i][1] = balls[1];
    entries[i][2] = balls[2];
    entries[i][3] = balls[3];
    entries[i][4] = balls[4];
    entries[i][5] = balls[5];
  }

  time_t start = time(NULL);

  // minimum values
  long minCost = LONG_MAX;
  int minCostMainBalls[MAIN_BALLS_SIZE];
  int minCostBonusBall = -1;

  // maximum values
  long maxCost = LONG_MIN;
  int maxCostMainBalls[MAIN_BALLS_SIZE];
  int maxCostBonusBall = -1;

  // iterate over main ball combinations
  int mainBalls[MAIN_BALLS_SIZE];
  for (mainBalls[0] = LOW_BALL; mainBalls[0] <= HIGH_BALL; mainBalls[0]++) {
    for (mainBalls[1] = mainBalls[0] + 1; mainBalls[1] <= HIGH_BALL; mainBalls[1]++) {
      for (mainBalls[2] = mainBalls[1] + 1; mainBalls[2] <= HIGH_BALL; mainBalls[2]++) {
        for (mainBalls[3] = mainBalls[2] + 1; mainBalls[3] <= HIGH_BALL; mainBalls[3]++) {
          for (mainBalls[4] = mainBalls[3] + 1; mainBalls[4] <= HIGH_BALL; mainBalls[4]++) {
            for (mainBalls[5] = mainBalls[4] + 1; mainBalls[5] <= HIGH_BALL; mainBalls[5]++) {

              // create bitmask for match lookup
              long mainBallsBitmask = bitmask(mainBalls, SIZEOF(mainBalls));

              // init match counts to zero
              int matchesCount[MATCHES_COUNT_SIZE] = {0, 0, 0, 0, 0, 0, 0, 0};

              //init bonus ball
              int bonusBall = -1;

              // iterate over entries
              int i;
              for (i = 0; i < ENTRIES_SIZE; i++) {

                // count matches
                int m = matches(entries[i], mainBallsBitmask);

                // if 5 matches check the bonus ball
                if (m == MATCH_5) {

                  // iterate over bonus balls
                  for (bonusBall = LOW_BALL; bonusBall <= HIGH_BALL; bonusBall++) {

                    // skip bonus ball if its in the main balls
                    if (bitSet(mainBallsBitmask, bonusBall)) {
                      continue;
                    }

                    long entryBitmask = bitmask(entries[i], SIZEOF(entries));

                    // break if the entries contain the bonus ball
                    if (bitSet(entryBitmask, bonusBall)) {
                      m = MATCH_5_PLUS_BONUS;
                      break;
                    }
                  }
                }

                // increment matches for number of balls matched
                matchesCount[m]++;
              }

              // get total cost of draw
              long c = cost(matchesCount);

              // keep track of highest/lowest draw costs
              if (c < minCost) {
                minCost = c;
                memcpy(minCostMainBalls, mainBalls, MAIN_BALLS_SIZE * sizeof (int));
                minCostBonusBall = bonusBall;
              } else if (c > maxCost) {
                maxCost = c;
                memcpy(maxCostMainBalls, mainBalls, MAIN_BALLS_SIZE * sizeof (int));
                maxCostBonusBall = bonusBall;
              }
            }
          }
        }
      }
    }
  }

  time_t end = time(NULL);

  print(start, end, minCost, minCostMainBalls, minCostBonusBall,
          maxCost, maxCostMainBalls, maxCostBonusBall);

  return (EXIT_SUCCESS);
}

void shuffle(int a[], int size) {
  srand(time(NULL));
  int i;
  for (i = 0; i < size; i++) {
    int index = rand() % (i + 1);
    int x = a[index];
    a[index] = a[i];
    a[i] = x;
  }
}

int bitSet(long bitMask, int i) {
  return (bitMask & (1L << i)) > 0;
}

long setBit(long bitmask, int i) {
  return bitmask | (1L << i);
}

long bitmask(int a[], int size) {

  long bitmask = 0;

  int i;
  for (i = 0; i < size; i++) {
    bitmask = setBit(bitmask, a[i]);
  }

  return bitmask;
}

int matches(int entry[ENTRY_BALLS_SIZE], long mainBallsBitmask) {

  int matches = 0;

  int i;
  for (i = 0; i < ENTRY_BALLS_SIZE; i++) {

    if (bitSet(mainBallsBitmask, entry[i])) {
      matches++;
    }
  }

  return matches;
}

long cost(int matchesCount[MATCHES_COUNT_SIZE]) {

  long cost = 0;

  // add jackpot prize
  if (matchesCount[MATCH_6] > 0) {
    cost = MATCH_6_PRIZE;
  }

  // add lesser prizes, multiplied by number of winners
  cost += matchesCount[MATCH_5_PLUS_BONUS] * MATCH_5_PLUS_BONUS_PRIZE
          + matchesCount[MATCH_5] * MATCH_5_PRIZE
          + matchesCount[MATCH_4] * MATCH_4_PRIZE
          + matchesCount[MATCH_3] * MATCH_3_PRIZE;

  return cost;
}

void print(
        time_t start, time_t end,
        long minCost, int minCostMainBalls[MAIN_BALLS_SIZE], int minCostBonusBall,
        long maxCost, int maxCostMainBalls[MAIN_BALLS_SIZE], int maxCostBonusBall) {

  printf("\n--- entries ---\n");
  printf("random entries processed = %d\n", ENTRIES_SIZE);

  printf("\n--- time ---\n");
  printf("elapsed time = %lu seconds\n", (end - start));

  printf("\n--- min ---\n");
  printf("minimum cost = %lu\n", minCost);
  printf("minimum main balls = %d,%d,%d,%d,%d,%d\n",
          minCostMainBalls[0],
          minCostMainBalls[1],
          minCostMainBalls[2],
          minCostMainBalls[3],
          minCostMainBalls[4],
          minCostMainBalls[5]);
  printf("minimum bonus ball = %d\n", minCostBonusBall);

  printf("\n--- max ---\n");
  printf("maximum cost = %lu\n", maxCost);
  printf("maximum main balls = %d,%d,%d,%d,%d,%d\n",
          maxCostMainBalls[0],
          maxCostMainBalls[1],
          maxCostMainBalls[2],
          maxCostMainBalls[3],
          maxCostMainBalls[4],
          maxCostMainBalls[5]);
  printf("maximum bonus ball = %d\n", maxCostBonusBall);
}

Output from the above C code:

--- entries ---
random entries processed = 10

--- time ---
elapsed time = 2 seconds

--- min ---
minimum cost = 0
minimum main balls = 1,2,3,4,5,9
minimum bonus ball = -1

--- max ---
maximum cost = 5000050
maximum main balls = 1,4,7,10,21,30
maximum bonus ball = -1

Twice as fast as the Java version for ten entries! Lets up the number of entries to 1000 to get a better idea of performance:

--- entries ---
random entries processed = 1000

--- time ---
elapsed time = 155 seconds

--- min ---
minimum cost = 0
minimum main balls = 1,2,7,9,10,11
minimum bonus ball = -1

--- max ---
maximum cost = 6101000
maximum main balls = 3,4,5,32,34,36
maximum bonus ball = 1

Processing 20,000,000 total entry lines should take 20,000 times longer than the previous output:

155 seconds * 20,000 = 3,100,000 seconds

3,100,000 seconds / (60 * 60 * 24) ~ 36 days

Even with the code optimisations and porting to C, 36 days is way longer than our 90 minute window.

Rigging the National Lottery is hard.

There will be further optimisations to be had with this code, but who cares, we all know how the Lottery is really rigged...

... MAGNETS!

Source Code

  • Java and C code available in GitHub - lotto

eCoster.co.uk – free recipe costing website

$
0
0

eCoster.co.uk is a free and simple recipe costing website for calculating a break down of what your meals cost.

eCoster.co.uk features:-

  • Recipe cost calculation - total cost and per serving cost.



  • Ingredients - unlimited ingredients, amount and cost entry.



  • Unit Conversions - standard weights and measures conversions, and also fully customisable.



  • Units - Use the default units or define you own custom measures.

eCoster is free to register and use, and the program source code is free to download and modify.

Source Code

Database Backed Map

$
0
0

I wanted an RDBMS backed Map, with it's keys and values immediately persisted to a relational database for use with a properties design pattern for prototype based programming.

I'm not sure if it'll be useful to anybody else, or even me for that matter, but I thought I'd put it out there anyway. The implementation, RdbmsMap, implements the java.util.Map interface and reads from, and writes to, a PostgreSQL database. The implementation supports key and values types of: null, Boolean, Integer, Double, String and RdbmsMap(the persistent Map class itself).

The ER diagram looks like this:


(Click To Enlarge)

The PostgreSQL code to create this schema is:

rdbms-map.sql

CREATE TABLE map
(
  id serial NOT NULL,
  CONSTRAINT map_pkey PRIMARY KEY (id)
);

CREATE TABLE entry
(
  id serial NOT NULL,
  map_id integer NOT NULL,
  key_type character(1) NOT NULL,
  value_type character(1) NOT NULL,
  CONSTRAINT entry_pkey PRIMARY KEY (id),
  CONSTRAINT entry_map_id_fkey FOREIGN KEY (map_id) REFERENCES map (id) ON DELETE CASCADE
);

CREATE INDEX entry_key_type_idx ON entry (key_type);
CREATE INDEX entry_value_type_idx ON entry (value_type);

CREATE TABLE object_integer
(
  id serial NOT NULL,
  entry_id integer NOT NULL,
  map_id integer NOT NULL,
  type character(1) NOT NULL,
  value integer NOT NULL,
  CONSTRAINT object_integer_pkey PRIMARY KEY (id),
  CONSTRAINT object_integer_entry_id_fkey FOREIGN KEY (entry_id) REFERENCES entry (id) ON DELETE CASCADE,
  CONSTRAINT object_integer_map_id_fkey FOREIGN KEY (map_id) REFERENCES map (id) ON DELETE CASCADE
);

CREATE INDEX object_integer_type_idx ON object_integer (type);
CREATE INDEX object_integer_value_idx ON object_integer (value);


CREATE TABLE object_boolean
(
  id serial NOT NULL,
  entry_id integer NOT NULL,
  map_id integer NOT NULL,
  type character(1) NOT NULL,
  value boolean NOT NULL,
  CONSTRAINT object_boolean_pkey PRIMARY KEY (id),
  CONSTRAINT object_boolean_entry_id_fkey FOREIGN KEY (entry_id) REFERENCES entry (id) ON DELETE CASCADE,
  CONSTRAINT object_boolean_map_id_fkey FOREIGN KEY (map_id) REFERENCES map (id) ON DELETE CASCADE
);

CREATE INDEX object_boolean_type_idx ON object_boolean (type);
CREATE INDEX object_boolean_value_idx ON object_boolean (value);

CREATE TABLE object_numeric
(
  id serial NOT NULL,
  entry_id integer NOT NULL,
  map_id integer NOT NULL,
  type character(1) NOT NULL,
  value numeric NOT NULL,
  CONSTRAINT object_numeric_pkey PRIMARY KEY (id),
  CONSTRAINT object_numeric_entry_id_fkey FOREIGN KEY (entry_id) REFERENCES entry (id) ON DELETE CASCADE,
  CONSTRAINT object_numeric_map_id_fkey FOREIGN KEY (map_id) REFERENCES map (id) ON DELETE CASCADE
);

CREATE INDEX object_numeric_type_idx ON object_numeric (type);
CREATE INDEX object_numeric_value_idx ON object_numeric (value);

CREATE TABLE object_text
(
  id serial NOT NULL,
  entry_id integer NOT NULL,
  map_id integer NOT NULL,
  type character(1) NOT NULL,
  value text NOT NULL,
  CONSTRAINT object_text_pkey PRIMARY KEY (id),
  CONSTRAINT object_text_entry_id_fkey FOREIGN KEY (entry_id) REFERENCES entry (id) ON DELETE CASCADE,
  CONSTRAINT object_text_map_id_fkey FOREIGN KEY (map_id) REFERENCES map (id) ON DELETE CASCADE
);

CREATE INDEX object_text_type_idx ON object_text (type);
CREATE INDEX object_text_value_idx ON object_text (value);

CREATE TABLE object_null
(
  id serial NOT NULL,
  entry_id integer NOT NULL,
  map_id integer NOT NULL,
  type character(1) NOT NULL,
  CONSTRAINT object_null_pkey PRIMARY KEY (id),
  CONSTRAINT object_null_entry_id_fkey FOREIGN KEY (entry_id) REFERENCES entry (id) ON DELETE CASCADE,
  CONSTRAINT object_null_map_id_fkey FOREIGN KEY (map_id) REFERENCES map (id) ON DELETE CASCADE
);

CREATE INDEX object_null_type_idx ON object_null (type);

CREATE TABLE object_map
(
  id serial NOT NULL,
  entry_id integer NOT NULL,
  map_id integer NOT NULL,
  type character(1) NOT NULL,
  value integer NOT NULL,
  CONSTRAINT object_map_pkey PRIMARY KEY (id),
  CONSTRAINT object_map_entry_id_fkey FOREIGN KEY (entry_id) REFERENCES entry (id) ON DELETE CASCADE,
  CONSTRAINT object_map_map_id_fkey FOREIGN KEY (map_id) REFERENCES map (id) ON DELETE CASCADE,
  CONSTRAINT object_map_value_fkey FOREIGN KEY (value) REFERENCES map (id) ON DELETE CASCADE
);

CREATE INDEX object_map_type_idx ON object_map (type);

The Map implementation is:

RdbmsMap.java

package org.adrianwalker.rdbmsmap;

import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.AbstractMap;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;

public final class RdbmsMap<K, V> implements Map<K, V> {

  // object types
  private static final String OBJECT_NULL_TYPE = "0";
  private static final String OBJECT_INTEGER_TYPE = "I";
  private static final String OBJECT_BOOLEAN_TYPE = "B";
  private static final String OBJECT_NUMERIC_TYPE = "N";
  private static final String OBJECT_TEXT_TYPE = "T";
  private static final String OBJECT_MAP_TYPE = "M";
  // entry types
  private static final String ENTRY_KEY_TYPE = "K";
  private static final String ENTRY_VALUE_TYPE = "V";
  // object types
  private static final String OBJECT_TABLE = "object_table";
  private static final String OBJECT_NULL_TABLE = "object_null";
  private static final String OBJECT_INTEGER_TABLE = "object_integer";
  private static final String OBJECT_BOOLEAN_TABLE = "object_boolean";
  private static final String OBJECT_NUMERIC_TABLE = "object_numeric";
  private static final String OBJECT_TEXT_TABLE = "object_text";
  private static final String OBJECT_MAP_TABLE = "object_map";
  // inserts
  private static final String INSERT_MAP = "insert into map(id) values(nextval('map_id_seq')) returning id";
  private static final String INSERT_ENTRY = "insert into entry(id, map_id, key_type, value_type) values(nextval('entry_id_seq'), ?, ?, ?) returning id";
  private static final String INSERT_OBJECT = "insert into " + OBJECT_TABLE + "(id, entry_id, map_id, type, value) values(nextval('" + OBJECT_TABLE + "_id_seq'), ?, ?, ?, ?) returning id";
  // counts
  private static final String COUNT_ENTRIES = "select count(*) from entry where map_id = ?";
  private static final String COUNT_OBJECT = "select count(*) from " + OBJECT_TABLE + " where map_id = ? and type = ? and value = ?";
  // selects
  private static final String SELECT_ENTRIES = "select id, key_type, value_type from entry where map_id = ?";
  private static final String SELECT_ENTRY_BY_OBJECT = "select value_type, id from entry where id = (select entry_id from " + OBJECT_TABLE + " where map_id = ? and type = ? and value = ?)";
  private static final String SELECT_OBJECT_BY_ENTRY = "select value from " + OBJECT_TABLE + " where map_id = ? and type = ? and entry_id = ?";
  private static final String SELECT_OBJECT = "select value from " + OBJECT_TABLE + " where map_id = ? and type = ?";
  // deletes
  private static final String DELETE_ENTRIES = "delete from entry where map_id = ?";
  private static final String DELETE_ENTRY_BY_OBJECT = "delete from entry where id = (select entry_id from " + OBJECT_TABLE + " where map_id = ? and type = ? and value = ?)";
  // specific cases for nulls
  private static final String INSERT_OBJECT_NULL = "insert into " + OBJECT_NULL_TABLE + "(id, entry_id, map_id, type) values(nextval('" + OBJECT_NULL_TABLE + "_id_seq'), ?, ?, ?) returning id";
  private static final String COUNT_OBJECT_NULL = "select count(*) from " + OBJECT_NULL_TABLE + " where map_id = ? and type = ?";
  private static final String SELECT_ENTRY_BY_OBJECT_NULL = "select value_type, id from entry where id = (select entry_id from " + OBJECT_NULL_TABLE + " where map_id = ? and type = ?)";
  private static final String SELECT_OBJECT_NULL_BY_ENTRY = "select null from " + OBJECT_NULL_TABLE + " where map_id = ? and type = ? and entry_id = ?";
  private static final String SELECT_OBJECT_NULL = "select null from " + OBJECT_NULL_TABLE + " where map_id = ? and type = ?";
  private static final String DELETE_ENTRY_BY_OBJECT_NULL = "delete from entry where id = (select entry_id from " + OBJECT_NULL_TABLE + " where map_id = ? and type = ?)";

  private final Connection connection;
  private final int mapId;

  public RdbmsMap(final Connection connection) {

    this.connection = connection;

    try {
      this.mapId = inserMap();

    } catch (final SQLException sqle) {
      throw new RuntimeException(sqle);
    }
  }

  private RdbmsMap(final Connection connection, final int mapId) {

    this.connection = connection;
    this.mapId = mapId;
  }

  public int getMapId() {
    return mapId;
  }

  @Override
  public void clear() {

    try {
      delete();
    } catch (final SQLException sqle) {
      throw new RuntimeException(sqle);
    }
  }

  @Override
  public boolean containsKey(final Object key) {

    try {
      return countObjects(key, ENTRY_KEY_TYPE) > 0;
    } catch (final SQLException sqle) {
      throw new RuntimeException(sqle);
    }
  }

  @Override
  public boolean containsValue(final Object value) {

    try {
      return countObjects(value, ENTRY_VALUE_TYPE) > 0;
    } catch (final SQLException sqle) {
      throw new RuntimeException(sqle);
    }
  }

  @Override
  public Set<Entry<K, V>> entrySet() {

    try {
      return selectEntries();
    } catch (final SQLException sqle) {
      throw new RuntimeException(sqle);
    }
  }

  @Override
  public V get(final Object key) {

    try {
      return select(key);
    } catch (final SQLException sqle) {
      throw new RuntimeException(sqle);
    }
  }

  @Override
  public Set<K> keySet() {

    try {
      return (Set<K>) selectObjects(ENTRY_KEY_TYPE);
    } catch (final SQLException sqle) {
      throw new RuntimeException(sqle);
    }
  }

  @Override
  public V put(final K key, final V value) {

    Object previousValue = remove(key);

    try {
      insert(key, value);
    } catch (final SQLException sqle) {
      throw new RuntimeException(sqle);
    }

    return (V) previousValue;
  }

  @Override
  public void putAll(final Map<? extends K, ? extends V> m) {

    for (Entry<? extends K, ? extends V> entry : m.entrySet()) {
      put(entry.getKey(), entry.getValue());
    }
  }

  @Override
  public V remove(final Object key) {

    try {
      if (countObjects(key, ENTRY_KEY_TYPE) > 0) {
        V value = select(key);
        delete(key, ENTRY_KEY_TYPE);

        return value;
      }
    } catch (final SQLException sqle) {
      throw new RuntimeException(sqle);
    }

    return null;
  }

  @Override
  public int size() {

    try {
      return countEntries();
    } catch (final SQLException sqle) {
      throw new RuntimeException(sqle);
    }
  }

  @Override
  public boolean isEmpty() {

    return size() == 0;
  }

  @Override
  public Collection<V> values() {

    try {
      return selectObjects(ENTRY_VALUE_TYPE);
    } catch (final SQLException sqle) {
      throw new RuntimeException(sqle);
    }
  }

  private PreparedStatement prepareStatement(final String sql) throws SQLException {

    return connection.prepareStatement(sql);
  }

  private int inserMap() throws SQLException {

    PreparedStatement insertMap = prepareStatement(INSERT_MAP);

    ResultSet result = insertMap.executeQuery();
    if (!result.next()) {
      throw new RuntimeException();
    }

    return result.getInt(1);
  }

  private int countEntries() throws SQLException {

    PreparedStatement countEntries = prepareStatement(COUNT_ENTRIES);
    countEntries.setInt(1, mapId);

    ResultSet result = countEntries.executeQuery();
    if (!result.next()) {
      return 0;
    }

    return result.getInt(1);
  }

  private int countObjects(final Object obj, final String entryType) throws SQLException {

    String objectType = getType(obj);

    PreparedStatement countObject;

    if (objectType.equals(OBJECT_NULL_TYPE)) {
      countObject = prepareStatement(COUNT_OBJECT_NULL);
    } else if (objectType.equals(OBJECT_INTEGER_TYPE)) {
      countObject = prepareStatement(COUNT_OBJECT.replace(OBJECT_TABLE, OBJECT_INTEGER_TABLE));
      countObject.setInt(3, (Integer) obj);
    } else if (objectType.equals(OBJECT_BOOLEAN_TYPE)) {
      countObject = prepareStatement(COUNT_OBJECT.replace(OBJECT_TABLE, OBJECT_BOOLEAN_TABLE));
      countObject.setBoolean(3, (Boolean) obj);
    } else if (objectType.equals(OBJECT_NUMERIC_TYPE)) {
      countObject = prepareStatement(COUNT_OBJECT.replace(OBJECT_TABLE, OBJECT_NUMERIC_TABLE));
      countObject.setDouble(3, (Double) obj);
    } else if (objectType.equals(OBJECT_TEXT_TYPE)) {
      countObject = prepareStatement(COUNT_OBJECT.replace(OBJECT_TABLE, OBJECT_TEXT_TABLE));
      countObject.setString(3, (String) obj);
    } else if (objectType.equals(OBJECT_MAP_TYPE)) {
      countObject = prepareStatement(COUNT_OBJECT.replace(OBJECT_TABLE, OBJECT_MAP_TABLE));
      countObject.setString(3, (String) obj);
    } else {
      return 0;
    }

    countObject.setInt(1, mapId);
    countObject.setString(2, entryType);
    countObject.executeQuery();

    ResultSet result = countObject.executeQuery();
    if (!result.next()) {
      return 0;
    }

    return result.getInt(1);
  }

  private V select(final Object key) throws SQLException {

    String keyType = getType(key);

    PreparedStatement selectEntry;

    if (keyType.equals(OBJECT_NULL_TYPE)) {
      selectEntry = prepareStatement(SELECT_ENTRY_BY_OBJECT_NULL);
    } else if (keyType.equals(OBJECT_INTEGER_TYPE)) {
      selectEntry = prepareStatement(SELECT_ENTRY_BY_OBJECT.replace(OBJECT_TABLE, OBJECT_INTEGER_TABLE));
      selectEntry.setInt(3, (Integer) key);
    } else if (keyType.equals(OBJECT_BOOLEAN_TYPE)) {
      selectEntry = prepareStatement(SELECT_ENTRY_BY_OBJECT.replace(OBJECT_TABLE, OBJECT_BOOLEAN_TABLE));
      selectEntry.setBoolean(3, (Boolean) key);
    } else if (keyType.equals(OBJECT_NUMERIC_TYPE)) {
      selectEntry = prepareStatement(SELECT_ENTRY_BY_OBJECT.replace(OBJECT_TABLE, OBJECT_NUMERIC_TABLE));
      selectEntry.setDouble(3, (Double) key);
    } else if (keyType.equals(OBJECT_TEXT_TYPE)) {
      selectEntry = prepareStatement(SELECT_ENTRY_BY_OBJECT.replace(OBJECT_TABLE, OBJECT_TEXT_TABLE));
      selectEntry.setString(3, (String) key);
    } else if (keyType.equals(OBJECT_MAP_TYPE)) {
      selectEntry = prepareStatement(SELECT_ENTRY_BY_OBJECT.replace(OBJECT_TABLE, OBJECT_MAP_TABLE));
      selectEntry.setString(3, (String) key);
    } else {
      return null;
    }

    selectEntry.setInt(1, mapId);
    selectEntry.setString(2, ENTRY_KEY_TYPE);

    ResultSet result = selectEntry.executeQuery();
    if (!result.next()) {
      return null;
    }

    String objectType = result.getString(1);
    int entryId = result.getInt(2);

    Object value = selectObject(objectType, ENTRY_VALUE_TYPE, entryId);

    return (V) value;
  }

  private Collection selectObjects(final String entryType, final String objectType) throws SQLException {

    Collection objs;

    if (entryType.equals(ENTRY_KEY_TYPE)) {
      objs = new HashSet();
    } else if (entryType.equals(ENTRY_VALUE_TYPE)) {
      objs = new ArrayList();
    } else {
      return null;
    }

    PreparedStatement selectObject;

    if (objectType.equals(OBJECT_NULL_TYPE)) {
      selectObject = prepareStatement(SELECT_OBJECT_NULL);
    } else if (objectType.equals(OBJECT_INTEGER_TYPE)) {
      selectObject = prepareStatement(SELECT_OBJECT.replace(OBJECT_TABLE, OBJECT_INTEGER_TABLE));
    } else if (objectType.equals(OBJECT_BOOLEAN_TYPE)) {
      selectObject = prepareStatement(SELECT_OBJECT.replace(OBJECT_TABLE, OBJECT_BOOLEAN_TABLE));
    } else if (objectType.equals(OBJECT_NUMERIC_TYPE)) {
      selectObject = prepareStatement(SELECT_OBJECT.replace(OBJECT_TABLE, OBJECT_NUMERIC_TABLE));
    } else if (objectType.equals(OBJECT_TEXT_TYPE)) {
      selectObject = prepareStatement(SELECT_OBJECT.replace(OBJECT_TABLE, OBJECT_TEXT_TABLE));
    } else if (objectType.equals(OBJECT_MAP_TYPE)) {
      selectObject = prepareStatement(SELECT_OBJECT.replace(OBJECT_TABLE, OBJECT_MAP_TABLE));
    } else {
      return objs;
    }

    selectObject.setInt(1, mapId);
    selectObject.setString(2, entryType);

    ResultSet result = selectObject.executeQuery();

    while (result.next()) {
      if (objectType.equals(OBJECT_NULL_TYPE)) {
        objs.add(null);
      } else if (objectType.equals(OBJECT_INTEGER_TYPE)) {
        objs.add(result.getInt(1));
      } else if (objectType.equals(OBJECT_BOOLEAN_TYPE)) {
        objs.add(result.getBoolean(1));
      } else if (objectType.equals(OBJECT_NUMERIC_TYPE)) {
        objs.add(result.getDouble(1));
      } else if (objectType.equals(OBJECT_TEXT_TYPE)) {
        objs.add(result.getString(1));
      } else if (objectType.equals(OBJECT_MAP_TYPE)) {
        objs.add(new RdbmsMap<K, V>(connection, result.getInt(1)));
      }
    }

    return objs;
  }

  private Collection selectObjects(final String entryType) throws SQLException {

    Collection objs = selectObjects(entryType, OBJECT_NULL_TYPE);
    objs.addAll(selectObjects(entryType, OBJECT_INTEGER_TYPE));
    objs.addAll(selectObjects(entryType, OBJECT_BOOLEAN_TYPE));
    objs.addAll(selectObjects(entryType, OBJECT_NUMERIC_TYPE));
    objs.addAll(selectObjects(entryType, OBJECT_TEXT_TYPE));
    objs.addAll(selectObjects(entryType, OBJECT_MAP_TYPE));

    return objs;
  }

  private String getType(final Object obj) {

    String keyType;

    if (null == obj) {
      keyType = OBJECT_NULL_TYPE;
    } else if (obj instanceof Integer) {
      keyType = OBJECT_INTEGER_TYPE;
    } else if (obj instanceof Boolean) {
      keyType = OBJECT_BOOLEAN_TYPE;
    } else if (obj instanceof Number) {
      keyType = OBJECT_NUMERIC_TYPE;
    } else if (obj instanceof String) {
      keyType = OBJECT_TEXT_TYPE;
    } else if (obj instanceof RdbmsMap) {
      keyType = OBJECT_MAP_TYPE;
    } else {
      return null;
    }

    return keyType;
  }

  private int insertEntry(final K key, final V value) throws SQLException {

    String keyType = getType(key);
    String valueType = getType(value);

    PreparedStatement insertEntry = prepareStatement(INSERT_ENTRY);
    insertEntry.setInt(1, mapId);
    insertEntry.setString(2, keyType);
    insertEntry.setString(3, valueType);

    ResultSet result = insertEntry.executeQuery();
    if (!result.next()) {
      return 0;
    }

    return result.getInt(1);
  }

  private void insertObject(final int entryId, final Object obj, final String entryType) throws SQLException {

    String objectType = getType(obj);

    PreparedStatement insertObject;

    if (objectType.equals(OBJECT_NULL_TYPE)) {
      insertObject = prepareStatement(INSERT_OBJECT_NULL);
    } else if (objectType.equals(OBJECT_INTEGER_TYPE)) {
      insertObject = prepareStatement(INSERT_OBJECT.replace(OBJECT_TABLE, OBJECT_INTEGER_TABLE));
      insertObject.setInt(4, (Integer) obj);
    } else if (objectType.equals(OBJECT_BOOLEAN_TYPE)) {
      insertObject = prepareStatement(INSERT_OBJECT.replace(OBJECT_TABLE, OBJECT_BOOLEAN_TABLE));
      insertObject.setBoolean(4, (Boolean) obj);
    } else if (objectType.equals(OBJECT_NUMERIC_TYPE)) {
      insertObject = prepareStatement(INSERT_OBJECT.replace(OBJECT_TABLE, OBJECT_NUMERIC_TABLE));
      insertObject.setDouble(4, (Double) obj);
    } else if (objectType.equals(OBJECT_TEXT_TYPE)) {
      insertObject = prepareStatement(INSERT_OBJECT.replace(OBJECT_TABLE, OBJECT_TEXT_TABLE));
      insertObject.setString(4, (String) obj);
    } else if (objectType.equals(OBJECT_MAP_TYPE)) {
      insertObject = prepareStatement(INSERT_OBJECT.replace(OBJECT_TABLE, OBJECT_MAP_TABLE));
      insertObject.setInt(4, ((RdbmsMap) obj).mapId);
    } else {
      return;
    }

    insertObject.setInt(1, entryId);
    insertObject.setInt(2, mapId);
    insertObject.setString(3, entryType);
    insertObject.executeQuery();
  }

  private void insert(final K key, final V value) throws SQLException {

    int entryId = insertEntry(key, value);
    insertObject(entryId, key, ENTRY_KEY_TYPE);
    insertObject(entryId, value, ENTRY_VALUE_TYPE);
  }

  private void delete(final Object obj, final String entryType) throws SQLException {

    String objectType = getType(obj);

    PreparedStatement deleteEntry;
    if (objectType.equals(OBJECT_NULL_TYPE)) {
      deleteEntry = prepareStatement(DELETE_ENTRY_BY_OBJECT_NULL);
    } else if (objectType.equals(OBJECT_INTEGER_TYPE)) {
      deleteEntry = prepareStatement(DELETE_ENTRY_BY_OBJECT.replace(OBJECT_TABLE, OBJECT_INTEGER_TABLE));
      deleteEntry.setInt(3, (Integer) obj);
    } else if (objectType.equals(OBJECT_BOOLEAN_TYPE)) {
      deleteEntry = prepareStatement(DELETE_ENTRY_BY_OBJECT.replace(OBJECT_TABLE, OBJECT_BOOLEAN_TABLE));
      deleteEntry.setBoolean(3, (Boolean) obj);
    } else if (objectType.equals(OBJECT_NUMERIC_TYPE)) {
      deleteEntry = prepareStatement(DELETE_ENTRY_BY_OBJECT.replace(OBJECT_TABLE, OBJECT_NUMERIC_TABLE));
      deleteEntry.setDouble(3, (Double) obj);
    } else if (objectType.equals(OBJECT_TEXT_TYPE)) {
      deleteEntry = prepareStatement(DELETE_ENTRY_BY_OBJECT.replace(OBJECT_TABLE, OBJECT_TEXT_TABLE));
      deleteEntry.setString(3, (String) obj);
    } else if (objectType.equals(OBJECT_MAP_TYPE)) {
      deleteEntry = prepareStatement(DELETE_ENTRY_BY_OBJECT.replace(OBJECT_TABLE, OBJECT_MAP_TABLE));
      deleteEntry.setInt(3, ((RdbmsMap) obj).mapId);
    } else {
      return;
    }

    deleteEntry.setInt(1, mapId);
    deleteEntry.setString(2, entryType);
    deleteEntry.executeUpdate();
  }

  private void delete() throws SQLException {

    PreparedStatement deleteEntries = prepareStatement(DELETE_ENTRIES);
    deleteEntries.setInt(1, mapId);
    deleteEntries.executeUpdate();
  }

  private Object selectObject(final String objectType, final String entryType, final int entryId) throws SQLException {

    PreparedStatement selectObject;
    if (objectType.equals(OBJECT_NULL_TYPE)) {
      selectObject = prepareStatement(SELECT_OBJECT_NULL_BY_ENTRY);
    } else if (objectType.equals(OBJECT_INTEGER_TYPE)) {
      selectObject = prepareStatement(SELECT_OBJECT_BY_ENTRY.replace(OBJECT_TABLE, OBJECT_INTEGER_TABLE));
    } else if (objectType.equals(OBJECT_BOOLEAN_TYPE)) {
      selectObject = prepareStatement(SELECT_OBJECT_BY_ENTRY.replace(OBJECT_TABLE, OBJECT_BOOLEAN_TABLE));
    } else if (objectType.equals(OBJECT_NUMERIC_TYPE)) {
      selectObject = prepareStatement(SELECT_OBJECT_BY_ENTRY.replace(OBJECT_TABLE, OBJECT_NUMERIC_TABLE));
    } else if (objectType.equals(OBJECT_TEXT_TYPE)) {
      selectObject = prepareStatement(SELECT_OBJECT_BY_ENTRY.replace(OBJECT_TABLE, OBJECT_TEXT_TABLE));
    } else if (objectType.equals(OBJECT_MAP_TYPE)) {
      selectObject = prepareStatement(SELECT_OBJECT_BY_ENTRY.replace(OBJECT_TABLE, OBJECT_MAP_TABLE));
    } else {
      return null;
    }

    selectObject.setInt(1, mapId);
    selectObject.setString(2, entryType);
    selectObject.setInt(3, entryId);

    ResultSet result = selectObject.executeQuery();

    if (!result.next()) {
      return null;
    }

    Object value;
    if (objectType.equals(OBJECT_NULL_TYPE)) {
      value = null;
    } else if (objectType.equals(OBJECT_INTEGER_TYPE)) {
      value = result.getInt(1);
    } else if (objectType.equals(OBJECT_BOOLEAN_TYPE)) {
      value = result.getBoolean(1);
    } else if (objectType.equals(OBJECT_NUMERIC_TYPE)) {
      value = result.getDouble(1);
    } else if (objectType.equals(OBJECT_TEXT_TYPE)) {
      value = result.getString(1);
    } else if (objectType.equals(OBJECT_MAP_TYPE)) {
      value = new RdbmsMap<K, V>(connection, result.getInt(1));
    } else {
      return null;
    }

    return value;
  }

  private Set<Entry<K, V>> selectEntries() throws SQLException {

    Set<Entry<K, V>> entries = new HashSet<Entry<K, V>>();

    PreparedStatement selectEntries = prepareStatement(SELECT_ENTRIES);
    selectEntries.setInt(1, mapId);

    ResultSet result = selectEntries.executeQuery();

    while (result.next()) {
      int entryId = result.getInt(1);
      String keyType = result.getString(2);
      String valueType = result.getString(3);

      Entry<K, V> entry = new AbstractMap.SimpleEntry<K, V>(
              (K) selectObject(keyType, ENTRY_KEY_TYPE, entryId),
              (V) selectObject(valueType, ENTRY_VALUE_TYPE, entryId));

      entries.add(entry);
    }

    return entries;
  }
}

For examples and tests check out the source code.

Source Code

Optimise Your Own Fuckin' Tail Calls

$
0
0

I've heard a few people moaning that Java 8 still doesn't optimise tail calls. Not having programmed in a functional language since Moscow ML at University, I didn't really give a shit, but thought I should find out what the deal is.

A tail call method is a method where the final action is to call another method. And a tail-recursive method is when a method's final action is to call itself.

Now, we all know recursion is mathematically beautiful and elegant for people who write programs that don't do anything. For the rest of us, using recursion for anything non-trivial means you're going to run out of stack at some point.

For example NASA's JPL C Coding Standard disallows the use of recursion because:-

"The presence of statically verifiable loop bounds and the absence of recursion prevent runaway code, and help to secure predictable performance for all tasks. The absence of recursion also simplifies the task of deriving reliable bounds on stack use. The two rules combined secure a strictly acyclic function call graph and control-flow structure, which in turn enhances the capabilities for static checking tools to catch a broad range of coding defects."

Languages which support tail call optimisation replace tail call recursion with a loop so you don't have to worry about runaway stack usage. Think about that - the compiler tries to get rid of recursion for you, because it's inefficient and prone to causing defects. How elegant is your recursive algorithm really?

Also, in what other situation would you expect the compiler to so drastically change your code for you? Yeah I know you can expect the Java JIT compiler to perform optimisations, but tail call elimination is a step too far for me. While you're optimising my recursive calls, why don't you just go ahead and write the rest of my algorithms for me? - GET OFF MY LAWN!

A good example of converting a recursive algorithm to a while loop is available here: http://c2.com/cgi/wiki?TailCallOptimization and I've converted it to Java below.

I've used BigInteger in the code to avoid integer overflow, because I want to demonstrate stack overflow with the default JVM settings.

First, a recursive method which is NOT tail call recursive. The method is not tail call recursive because the final action in the method is not a call to itself but a multiplication:

    private static BigInteger notTailRecursionFactorial(final BigInteger n) {

        if (n.compareTo(TWO) < 0) {
            return ONE;
        } else {
            return n.multiply(notTailRecursionFactorial(n.subtract(ONE)));
        }
    }

Modifying the method to use an accumulator variable makes the method tail call recursive, and a candidate for tail call optimisation:

    private static BigInteger tailRecursionFactorial(final BigInteger n, final BigInteger accumulator) {

        if (n.compareTo(TWO) < 0) {
            return accumulator;
        } else {
            return tailRecursionFactorial(n.subtract(ONE), n.multiply(accumulator));
        }
    }

The recursive method converted to a while loop:

    private static BigInteger tailRecursionEliminationFactorial(BigInteger n, BigInteger accumulator) {

        while (n.compareTo(TWO) >= 0) {
            accumulator = accumulator.multiply(n);
            n = n.subtract(ONE);
        }

        return accumulator;
    }

Further optimisation to the tail call eliminated method:

    private static BigInteger tailRecursionEliminationFactorialOptimised(BigInteger n) {

        BigInteger accumulator = ONE;

        while (n.compareTo(TWO) >= 0) {
            accumulator = accumulator.multiply(n);
            n = n.subtract(ONE);
        }

        return accumulator;
    }

The full class looks like this:

TailCallOptimisation.java

package org.adrianwalker;

import java.math.BigInteger;

public class TailCallOptimisation {

  private static final BigInteger ONE = new BigInteger("1");
  private static final BigInteger TWO = new BigInteger("2");
  private static final BigInteger ONE_HUNDRED_THOUSAND = new BigInteger("100000");

  public static void main(final String[] args) {

    System.out.println("\nnotTailRecursionFactorial:");

    try {
      System.out.println(notTailRecursionFactorial(ONE_HUNDRED_THOUSAND));
    } catch (final Throwable t) {
      System.out.println("Stack Overflow Error");
    }

    System.out.println("\ntailRecursionFactorial:");

    try {
      System.out.println(tailRecursionFactorial(ONE_HUNDRED_THOUSAND, ONE));
    } catch (final Throwable t) {
      System.out.println("Stack Overflow Error");
    }

    System.out.println("\ntailRecursionEliminationFactorial:");

    System.out.println(tailRecursionEliminationFactorial(ONE_HUNDRED_THOUSAND, ONE));

    System.out.println("\ntailRecursionEliminationFactorialOptimised:");

    System.out.println(tailRecursionEliminationFactorialOptimised(ONE_HUNDRED_THOUSAND));
  }

  private static BigInteger notTailRecursionFactorial(final BigInteger n) {

    if (n.compareTo(TWO) < 0) {
      return ONE;
    } else {
      return n.multiply(notTailRecursionFactorial(n.subtract(ONE)));
    }
  }

  private static BigInteger tailRecursionFactorial(final BigInteger n, final BigInteger accumulator) {

    if (n.compareTo(TWO) < 0) {
      return accumulator;
    } else {
      return tailRecursionFactorial(n.subtract(ONE), n.multiply(accumulator));
    }
  }

  private static BigInteger tailRecursionEliminationFactorial(BigInteger n, BigInteger accumulator) {

    while (n.compareTo(TWO) >= 0) {
      accumulator = accumulator.multiply(n);
      n = n.subtract(ONE);
    }

    return accumulator;
  }

  private static BigInteger tailRecursionEliminationFactorialOptimised(BigInteger n) {

    BigInteger accumulator = ONE;

    while (n.compareTo(TWO) >= 0) {
      accumulator = accumulator.multiply(n);
      n = n.subtract(ONE);
    }

    return accumulator;
  }
}

And the output on my machine is:

notTailRecursionFactorial:
Stack Overflow Error

tailRecursionFactorial:
Stack Overflow Error

tailRecursionEliminationFactorial:
282422940796034787429342157802453551847749492609... (loads more)

tailRecursionEliminationFactorialOptimised:
282422940796034787429342157802453551847749492609... (loads more)

The recursive methods run out of stack way before getting anywhere near an answer for 100,000!.

Bottom line - optimise your own fuckin' tail calls.

Update - 24/08/14

A clever chap called Dr Rowan Davies got in touch about this post. Here's some food for thought:-

Just a small comment that your page "Optimise Your Own Fuckin' Tail Calls" is missing the point.

Basically, you've only considered the very simplest situation, a single recursive function that corresponds to a loop. Functional compilers like Scala do exactly the transformation to loops you've shown, so no one is complaining about that kind of tail call on the JVM.

It is the much more powerful uses of tail calls for things that don't correspond to loops. E.g., in F# on .NET (which supports tail calls) there is really nice support for asynchronous programming that depends on tail calls to avoid the stack increasing when you swap between different asynchronous handlers and lightweight software threads. The correspond code in Scala can't do that, the JVM just doesn't support it - and I challenge you to convert such asynchronous code using tail calls to Java code that doesn't chew up stack as it goes.

And, optimized tail-calls are not rewriting your code - they are exactly doing the tail call, just in a way that doesn't keep crap on your stack that clearly can never be needed in the future. Basically rejecting tail call optimization is demanding that the JVM keep this unneeded crap on the stack. Why do you insist that the JVM keep crap around? It's equivalent to insisting that for loops chew up stack space as they go through iterations - it really makes no sense at all to require that.

And this one as well:-

Your view is actually pretty common. But, it really is unnatural to make the implementation of tail calls consume stack when they don't have to. It's not so much about mathematical elegance, it's about whether certain powerful and natural ways of programming explode the stack or not. You're expecting a function call to always be implemented in a certain way that is really non-optimal sometimes. It's not really an optimization, it's about not doing something stupid that keeps potentially huge amounts of unnecessary data on the stack. gcc has done this basically forever. Indeed, many years ago I checked the machine code output and the C compiler cc in UNIX System V and it clearly supported efficient tail calls back in 1989.

The JVM doesn't include efficient tail calls largely because the JVM security model isn't so compatible with them, and changing the model would change the assumptions existing code could depend on. This is a basically a bureaucratic reason, the requirement to support earlier JVM programs that depend on the lack of efficient tail calls. But, many hackers and language implementers will celebrate if it is allowed sometime, otherwise those hackers and languages will eventually move on to VMs that do support what they need.

Cheers for the comments doc.

Fundamental 2D Game Programming With Java Review

Lispy Java 8

$
0
0

After reading Peter Norvig's post (How to Write a (Lisp) Interpreter (in Python)), I thought I'd have a go at doing the same thing in Java 8:

Lispy.java

package org.adrianwalker.lispy;

import java.util.*;
import java.util.function.*;
import java.util.stream.*;

public final class Lispy {

  private class Env extends HashMap<String, Object> {

    private final Env outer;

    public Env(final List<String> params, final List<Object> args, final Env outer) {
      IntStream.range(0, params.size()).forEach(i -> put(params.get(i), args.get(i)));
      this.outer = outer;
    }

    public Env find(final Object var) {
      return this.containsKey(var) ? this : outer.find(var);
    }
  }

  private Env addGlobal(final Env env) {
    env.put("+", (Function<List<Double>, Double>) (List<Double> args) -> args.get(0) + args.get(1));
    env.put("-", (Function<List<Double>, Double>) (List<Double> args) -> args.get(0) - args.get(1));
    env.put("*", (Function<List<Double>, Double>) (List<Double> args) -> args.get(0) * args.get(1));
    env.put("/", (Function<List<Double>, Double>) (List<Double> args) -> args.get(0) / args.get(1));
    env.put(">", (Function<List<Double>, Boolean>) (List<Double> args) -> args.get(0) > args.get(1));
    env.put("<", (Function<List<Double>, Boolean>) (List<Double> args) -> args.get(0) < args.get(1));
    env.put(">=", (Function<List<Double>, Boolean>) (List<Double> args) -> args.get(0) >= args.get(1));
    env.put("<=", (Function<List<Double>, Boolean>) (List<Double> args) -> args.get(0) <= args.get(1));
    env.put("=", (Function<List<Double>, Boolean>) (List<Double> args) -> Objects.equals(args.get(0), args.get(1)));
    env.put("equal?", (Function<List<Double>, Boolean>) (List<Double> args) -> Objects.equals(args.get(0), args.get(1)));
    env.put("eq?", (Function<List<Object>, Boolean>) (List<Object> args) -> args.get(0).getClass().isInstance(args.get(1)));
    env.put("length", (Function<List<Object>, Integer>) (List<Object> args) -> args.size());
    env.put("cons", (Function<List<Object>, List<Object>>) (List<Object> args) -> Stream.concat(Stream.of(args.get(0)), Stream.of(args.get(1))).collect(Collectors.toList()));
    env.put("car", (Function<List<Object>, Object>) (List<Object> args) -> ((List) args.get(0)).get(0));
    env.put("cdr", (Function<List<Object>, List<Object>>) (List<Object> args) -> ((List) args.get(0)).subList(1, ((List) args.get(0)).size()));
    env.put("append", (Function<List<Object>, List<Object>>) (List<Object> args) -> (List) Stream.concat(((List) args.get(0)).stream(), ((List) args.get(1)).stream()).collect(Collectors.toList()));
    env.put("list", (Function<List<Object>, List<Object>>) (List<Object> args) -> args);
    env.put("list?", (Function<List<Object>, Boolean>) (List<Object> args) -> args.get(0) instanceof List);
    env.put("null?", (Function<List<Object>, Boolean>) (List<Object> args) -> ((List) args.get(0)).isEmpty());
    env.put("symbol?", (Function<List<Object>, Boolean>) (List<Object> args) -> args.get(0) instanceof String);
    return env;
  }

  private final Env globalEnv = addGlobal(new Env(Collections.EMPTY_LIST, Collections.EMPTY_LIST, null));

  private Object eval(final Object x, final Env env) {
    if (x instanceof String) {
      return env.find(x).get(x);
    } else if (!(x instanceof List)) {
      return x;
    } else if (((List) x).get(0).equals("quote")) {
      return ((List) x).get(1);
    } else if (((List) x).get(0).equals("if")) {
      return eval(((Boolean) eval(((List) x).get(1), env)) ? ((List) x).get(2) : ((List) x).get(3), env);
    } else if (((List) x).get(0).equals("set!")) {
      env.find(((List) x).get(1)).put((String) ((List) x).get(1), eval(((List) x).get(2), env));
    } else if (((List) x).get(0).equals("define")) {
      env.put((String) ((List) x).get(1), eval(((List) x).get(2), env));
    } else if (((List) x).get(0).equals("lambda")) {
      return (Function<List<Object>, Object>) (List<Object> args) -> eval(((List) x).get(2), new Env((List) ((List) x).get(1), args, env));
    } else if (((List) x).get(0).equals("begin")) {
      Object val = null;
      for (Object exp : ((List) x).subList(1, ((List) x).size())) {
        val = eval(exp, env);
      }
      return val;
    } else {
      List exps = (List) ((List) x).stream().map(exp -> eval(exp, env)).collect(Collectors.toList());
      Function proc = (Function) exps.get(0);
      return proc.apply(exps.subList(1, exps.size()));
    }
    return null;
  }

  private Object read(final String s) {
    return readFrom(new ArrayDeque<>(Arrays.asList(tokenize(s))));
  }

  private String[] tokenize(final String s) {
    return s.replace("(", " ( ").replace(")", " ) ").trim().split("\\s+");
  }

  private Object readFrom(final Deque<String> tokens) {

    if (tokens.isEmpty()) {
      throw new RuntimeException("unexpected EOF while reading");
    }

    String token = tokens.pop();
    if ("(".equals(token)) {
      List l = new ArrayList();
      while (!tokens.peek().equals(")")) {
        l.add(readFrom(tokens));
      }
      tokens.pop();
      return l;
    } else if (")".equals(token)) {
      throw new RuntimeException("unexpected )");
    } else {
      return atom(token);
    }
  }

  private Object atom(final String token) {
    try {
      return Double.valueOf(token);
    } catch (final NumberFormatException nfe) {
      return token;
    }
  }

  private String toString(final Object exp) {
    return exp instanceof List ? "(" + ((List) exp).stream().map(x -> toString(x)).collect(Collectors.joining("")) + ")" : exp.toString();
  }

  private void repl() {
    while (true) {
      System.out.print("lispy>");
      Object val = eval(read(new Scanner(System.in).nextLine()), globalEnv);
      if (null != val) {
        System.out.println(toString(val));
      }
    }
  }

  public static void main(final String[] args) {
    new Lispy().repl();
  }
}

Source Code

  • Code available in GitHub - lispy

Continued Fraction Database File System

$
0
0

After reading Joe Celko'sSQL for Smarties on representing hierarchies in SQL databases, I wanted to have a go at creating a database backed file system.

Representing a file system using the Adjacency List Model detailed in chapter 2 is probably good enough for this sort of task, but that's a bit dull, and the Nested Set Model from chapter 4 is a cool way of looking at trees which I would have never have thought of.

Almost immediately the Nested Set model starts to look like a bad idea for a file system due to the performance implications of inserting into the middle of the tree.

Chapter 5 introduces the use of rational numbers and the Nested Interval Model from Vadim Tropashko, and at this point I quickly began not to care.

Then I came across this short but sweet report by Dan Hazel, Using rational numbers to key nested sets.

The paper quickly covers the Nested Set Model as described by Celko and touches on the similarity of the method presented to that of an encoding by Tropashko.

The paper describes a method of using rational numbers as node keys, and a way of calculating their numerators and denominators using the node positions in the tree represented as a continued fraction.

I wasn't taught continued fractions at school, so I had to go brush up before understanding the paper. It's actually a very interesting topic and there are a few good YouTube videos which can bring you up to speed. I liked this one.

I wont describe Dan Hazels report any further as I can't do it justice – it's very good, go read it now, and come back for a look at my implementation using Java and PostgreSQL.

Welcome back. First I needed a way of converting from a continued fraction node path into a fraction numerator and denominator.

  
public static int[] fraction(final int[] c) {

  int[] f = F[0];

  for (int i = c.length - 1; i >= 0; i--) {
    f = add(F[1], invert(f));
    f = add(fraction(c[i], 1), invert(f));
  }

  return f;
}

This function takes an array of integers which represent the continued fraction node path and returns a fraction as a 2 element integer array, containing the fractions numerator and denominator.

This operation is reversible and the function below takes a fraction as an integer array numerator and denominator and returns a continued fraction.

  
public static int[] continued(final int[] f) {

  int[] a = f;

  List<Integer> c = new ArrayList<>();
  while (a[0] > 0) {

    int i = a[0] / a[1];
    c.add(i);

    a = invert(subtract(a, fraction(i, 1)));
    a = invert(subtract(a, F[1]));
  }

  return toArray(c);
}

These are the two most important functions in a class which contains other operations required to manipulate fractions.

Fraction.java

package org.adrianwalker.continuedfractions;

import java.math.BigDecimal;
import java.math.RoundingMode;
import java.util.ArrayList;
import java.util.List;

public final class Fraction {

  private static final int[][] F = {
    {0, 1},
    {1, 1}
  };
  private static final int PRECISION = 16;

  private Fraction() {
  }

  public static int[] fraction(final int n, final int d) {

    return new int[]{n, d};
  }

  public static int[] fraction(final int[] pf, final int c, final int[] spf) {

    return fraction(pf[0] + c * spf[0], pf[1] + c * spf[1]);
  }

  public static int[] fraction(final int[] c) {

    int[] f = F[0];

    for (int i = c.length - 1; i >= 0; i--) {
      f = add(F[1], invert(f));
      f = add(fraction(c[i], 1), invert(f));
    }

    return f;
  }

  public static int[] continued(final int[] f) {

    int[] a = f;

    List<Integer> c = new ArrayList<>();
    while (a[0] > 0) {

      int i = a[0] / a[1];
      c.add(i);

      a = invert(subtract(a, fraction(i, 1)));
      a = invert(subtract(a, F[1]));
    }

    return toArray(c);
  }

  public static BigDecimal decimal(final int[] f) {

    return BigDecimal.valueOf(f[0]).divide(BigDecimal.valueOf(f[1]), PRECISION, RoundingMode.HALF_DOWN);
  }

  public static int[] add(final int[] f1, final int[] f2) {

    return fraction((f1[0] * f2[1]) + (f2[0] * f1[1]), f2[1] * f1[1]);
  }

  public static int[] subtract(final int[] f1, final int[] f2) {

    return fraction((f1[0] * f2[1]) - (f2[0] * f1[1]), f1[1] * f2[1]);
  }

  public static int[] invert(final int[] f) {

    return fraction(f[1], f[0]);
  }

  private static int[] toArray(final List<Integer> l) {

    int[] a = new int[l.size()];

    for (int i = 0; i < a.length; i++) {
      a[i] = l.get(i);
    }

    return a;
  }
}

Next I needed a class to manipulate Matrices and perform the sub tree moving calculations as detailed in the paper.

Matrix.java

package org.adrianwalker.continuedfractions;

public final class Matrix {

  private Matrix() {
  }

  public static int[][] matrix(
          final int M00, final int M01,
          final int M10, final int M11) {

    return new int[][]{{M00, M01}, {M10, M11}};
  }

  public static int[][] multiply(final int[][] M1, final int[][] M2) {

    return matrix(
            M1[0][0] * M2[0][0] + M1[0][1] * M2[1][0],
            M1[0][0] * M2[0][1] + M1[0][1] * M2[1][1],
            M1[1][0] * M2[0][0] + M1[1][1] * M2[1][0],
            M1[1][0] * M2[0][1] + M1[1][1] * M2[1][1]
    );
  }

  public static int[][] invert(final int[][] M) {

    return matrix(-M[1][1], M[0][1], M[1][0], -M[0][0]);
  }

  public static int[][] moveSubtree(final int[][] p0, final int m, final int[][] p1, int n, final int[][] M) {

    return multiply(multiply(multiply(p1, matrix(1, 0, m - n, 1)), invert(p0)), M);
  }
}

With the basic calculations nailed, next up is the database. The schema is a single table which uses the real value of the rational representation of the node as the primary key (id). Along with the primary key, the fractions numerator and denominator are stored (nv and dv), then the id and numerator and dominator of the nodes next largest sibling in the tree.

The level of the node in the tree is also stored, simply to help limit the depth of tree queries later on. And also the name of the node – what will be the file or directory name is stored in the row, along with a file content oid.

I've chosen PostgreSQL for this implementation - it provides an in row BYTEA type for storing binary data, or an off table Binary Large Object storage mechanism, with data accessible by an OID key. I've gone with the OIDs as they provide better support for IO streams, but the code could be easily modified to support BYTEA columns, or any other BLOB storage mechanism you might want to use in your RDBMS implementation.

files.sql

CREATE TABLE files
(
  id numeric NOT NULL,
  nv integer NOT NULL,
  dv integer NOT NULL,
  sid numeric NOT NULL,
  snv integer NOT NULL,
  sdv integer NOT NULL,
  level integer NOT NULL,
  name character varying NOT NULL,
  content oid NOT NULL,
  CONSTRAINT files_pkey PRIMARY KEY (id)
);

CREATE INDEX files_name_idx ON files (name);
CREATE INDEX files_level_idx ON files (level);

Next the Java database access code to perform basic operations on the database. This module uses continued fraction paths as inputs to almost all the methods, and is intended for lower level access to files held in the database.

FilesDAO.java

package org.adrianwalker.continuedfractions.filesystem;

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.math.BigDecimal;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.List;
import static org.adrianwalker.continuedfractions.Fraction.decimal;
import static org.adrianwalker.continuedfractions.Fraction.fraction;
import static org.adrianwalker.continuedfractions.Matrix.matrix;
import static org.adrianwalker.continuedfractions.Matrix.moveSubtree;
import static org.adrianwalker.continuedfractions.filesystem.Path.parent;
import static org.adrianwalker.continuedfractions.filesystem.Path.sibling;
import org.postgresql.largeobject.LargeObject;
import org.postgresql.largeobject.LargeObjectManager;

public final class FilesDAO {

  private static final String WRITE
          = "insert into files (id, nv, dv, sid, snv, sdv, level, name, content) "
          + "values(?, ?, ?, ?, ?, ?, ?, ?, ?)";
  private static final String READ
          = "select id, nv, dv, sid, snv, sdv, level, name, content "
          + "from files "
          + "where id = ?";
  private static final String TREE
          = "select id, nv, dv, sid, snv, sdv, level, name, content "
          + "from files "
          + "where id >= ? "
          + "and id < (select sid from files where id = ?)";
  private static final String CHILDREN
          = TREE
          + " and level = ?";
  private static final String CHILD
          = CHILDREN
          + " and name = ?";
  private static final String LAST_CHILD
          = CHILDREN
          + " order by id desc "
          + "limit 1";
  private static final String RENAME
          = "update files "
          + "set name = ? "
          + "where id = ?";
  private static final String MOVE
          = "update files "
          + "set id = ?, nv = ?, dv = ?, sid = ?, snv = ?, sdv = ?, level = ? "
          + "where id = ?";
  private static final String REMOVE
          = "delete from files "
          + "where id >= ? "
          + "and id < (select sid from files where id = ?)";
  private static final String CLEAR = "delete from files";
  private static final String ORDER_BY_ID = " order by id";
  private static final String ORDER_BY_NAME = " order by name";

  private final Connection connection;
  private final LargeObjectManager lom;

  public FilesDAO(final Connection connection) throws SQLException {

    this.connection = connection;
    this.lom = ((org.postgresql.PGConnection) connection).getLargeObjectAPI();
  }

  private PreparedStatement prepareStatement(final String sql) throws SQLException {

    return connection.prepareStatement(sql);
  }

  public int clear() throws SQLException {

    PreparedStatement remove = prepareStatement(CLEAR);

    return remove.executeUpdate();
  }

  public File write(final String filename, final int... path) throws SQLException {

    int[] f = fraction(path);
    int[] sf = fraction(sibling(path));
    int level = path.length;

    BigDecimal id = decimal(f);
    BigDecimal sid = decimal(sf);
    long content = lom.createLO();

    PreparedStatement write = prepareStatement(WRITE);
    write.setBigDecimal(1, id);
    write.setInt(2, f[0]);
    write.setInt(3, f[1]);
    write.setBigDecimal(4, sid);
    write.setInt(5, sf[0]);
    write.setInt(6, sf[1]);
    write.setInt(7, level);
    write.setString(8, filename);
    write.setLong(9, content);

    write.executeUpdate();

    return new File(id, f[0], f[1], sid, sf[0], sf[1], level, filename, content);
  }

  public File read(final int... path) throws SQLException {

    BigDecimal id = decimal(fraction(path));

    PreparedStatement read = prepareStatement(READ);
    read.setBigDecimal(1, id);

    ResultSet rs = read.executeQuery();

    return toFile(rs);
  }

  public File[] children(final int... path) throws SQLException {

    BigDecimal id = decimal(fraction(path));

    PreparedStatement children = prepareStatement(CHILDREN + ORDER_BY_NAME);
    children.setBigDecimal(1, id);
    children.setBigDecimal(2, id);
    children.setInt(3, path.length + 1);

    return toFiles(children.executeQuery());
  }

  public File child(final String name, final int... path) throws SQLException {

    BigDecimal id = decimal(fraction(path));

    PreparedStatement child = prepareStatement(CHILD);
    child.setBigDecimal(1, id);
    child.setBigDecimal(2, id);
    child.setInt(3, path.length + 1);
    child.setString(4, name);

    return toFile(child.executeQuery());
  }

  public File lastChild(final int... path) throws SQLException {

    BigDecimal id = decimal(fraction(path));

    PreparedStatement lastChild = prepareStatement(LAST_CHILD);
    lastChild.setBigDecimal(1, id);
    lastChild.setBigDecimal(2, id);
    lastChild.setInt(3, path.length + 1);

    return toFile(lastChild.executeQuery());
  }

  public File[] tree(final int... path) throws SQLException {

    BigDecimal id = decimal(fraction(path));

    PreparedStatement tree = prepareStatement(TREE + ORDER_BY_ID);
    tree.setBigDecimal(1, id);
    tree.setBigDecimal(2, id);

    return toFiles(tree.executeQuery());
  }

  public int rename(final String filename, final int... path) throws SQLException {

    BigDecimal id = decimal(fraction(path));

    PreparedStatement rename = prepareStatement(RENAME);
    rename.setString(1, filename);
    rename.setBigDecimal(2, id);

    return rename.executeUpdate();
  }

  public int remove(final int... path) throws SQLException {

    for (File file : tree(path)) {
      lom.delete(file.getContent());
    }

    BigDecimal id = decimal(fraction(path));

    PreparedStatement remove = prepareStatement(REMOVE);
    remove.setBigDecimal(1, id);
    remove.setBigDecimal(2, id);

    return remove.executeUpdate();
  }

  public int[] move(final int[] from, final int[] to) throws SQLException, IOException {

    return moveCopy(MOVE, from, to);
  }

  public int[] copy(final int[] from, final int[] to) throws SQLException, IOException {

    return moveCopy(WRITE, from, to);
  }

  private int[] moveCopy(final String sql, final int[] from, final int[] to) throws SQLException, IOException {

    int[] p = parent(from);
    int[] pf0 = fraction(p);
    int[] psf0 = fraction(sibling(p));
    int[] pf1 = fraction(to);
    int[] psf1 = fraction(sibling(to));

    int m = 1;
    File lc = lastChild(to);
    if (null != lc) {
      m = (lc.getSnv() - pf1[0]) / psf1[0];
    }

    int n = from[from.length - 1];

    int[][] p0 = matrix(pf0[0], psf0[0], pf0[1], psf0[1]);
    int[][] p1 = matrix(pf1[0], psf1[0], pf1[1], psf1[1]);

    PreparedStatement move = prepareStatement(sql);

    for (File file : tree(from)) {

      int[][] M0 = matrix(file.getNv(), file.getSnv(), file.getDv(), file.getSdv());
      int[][] M1 = moveSubtree(p0, m, p1, n, M0);

      int[] f = fraction(M1[0][0], M1[1][0]);
      int[] sf = fraction(M1[0][1], M1[1][1]);
      BigDecimal id = decimal(f);
      BigDecimal sid = decimal(sf);
      int level = to.length + (file.getLevel() - p.length);

      move.setBigDecimal(1, id);
      move.setInt(2, f[0]);
      move.setInt(3, f[1]);
      move.setBigDecimal(4, sid);
      move.setInt(5, sf[0]);
      move.setInt(6, sf[1]);
      move.setInt(7, level);

      switch (sql) {

        case MOVE:
          move.setBigDecimal(8, file.getId());
          break;

        case WRITE:
          long oid = lom.createLO();

          move.setString(8, file.getName());
          move.setLong(9, oid);

          Stream.copy(getInputStream(file.getContent()), getOutputStream(oid));

          break;
      }

      move.addBatch();
    }

    return move.executeBatch();
  }

  public InputStream getInputStream(final long oid) throws SQLException {

    LargeObject obj = lom.open(oid, LargeObjectManager.READ);

    return obj.getInputStream();
  }

  public OutputStream getOutputStream(final long oid) throws SQLException {

    LargeObject obj = lom.open(oid, LargeObjectManager.WRITE);

    return obj.getOutputStream();
  }

  private File toFile(final ResultSet rs) throws SQLException {

    File file = null;

    if (rs.next()) {
      file = new File(rs.getBigDecimal(1), rs.getInt(2), rs.getInt(3),
              rs.getBigDecimal(4), rs.getInt(5), rs.getInt(6),
              rs.getInt(7), rs.getString(8), rs.getLong(9));
    }

    return file;
  }

  private File[] toFiles(final ResultSet rs) throws SQLException {

    List<File> l = new ArrayList<>();

    while (rs.next()) {
      l.add(new File(rs.getBigDecimal(1), rs.getInt(2), rs.getInt(3),
              rs.getBigDecimal(4), rs.getInt(5), rs.getInt(6),
              rs.getInt(7), rs.getString(8), rs.getLong(9)));
    }

    return l.toArray(new File[l.size()]);
  }
}

And finally a higher level API for dealing directly with file names and paths in a manner similar to how you might use them on a command line.

FileSystem.java

package org.adrianwalker.continuedfractions.filesystem;

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.sql.SQLException;
import static org.adrianwalker.continuedfractions.filesystem.Path.range;

public final class FileSystem {

  private static final String SEPERATOR = "/";
  private static final String EMPTY_STRING = "";
  private final FilesDAO dao;
  private final int[] rootPath;

  public FileSystem(final FilesDAO dao, final int... rootPath) throws FileSystemException {

    this.dao = dao;
    this.rootPath = rootPath;

    try {
      if (null == dao.read(rootPath)) {
        dao.write("", rootPath);
      }
    } catch (final SQLException sqle) {
      throw new FileSystemException(sqle);
    }
  }

  public int[] create(final String path) throws FileSystemException {

    try {
      return path(path, true);
    } catch (final SQLException sqle) {
      throw new FileSystemException(sqle);
    }
  }

  public File[] list(final String path) throws FileSystemException {

    try {
      return dao.children(path(path, false));
    } catch (final SQLException sqle) {
      throw new FileSystemException(sqle);
    }
  }

  public File[] tree(final String path) throws FileSystemException {

    try {
      return dao.tree(path(path, false));
    } catch (final SQLException sqle) {
      throw new FileSystemException(sqle);
    }
  }

  public void write(final String path, final String text) throws FileSystemException {

    OutputStream out = getOutputStream(path);

    try {
      Stream.fromString(text, out);
    } catch (final IOException ioe) {
      throw new FileSystemException(ioe);
    }
  }

  public OutputStream getOutputStream(final String path) throws FileSystemException {

    OutputStream out;

    try {
      out = dao.getOutputStream(dao.read(path(path, true)).getContent());
    } catch (final SQLException sqle) {
      throw new FileSystemException(sqle);
    }

    return out;
  }

  public String read(final String path) throws FileSystemException {

    InputStream in = getInputStream(path);

    if (null == in) {
      return EMPTY_STRING;
    }

    try {
      return Stream.toString(in);
    } catch (final IOException ioe) {
      throw new FileSystemException(ioe);
    }
  }

  public InputStream getInputStream(final String path) throws FileSystemException {

    InputStream in;

    try {
      in = dao.getInputStream(dao.read(path(path, false)).getContent());
    } catch (final SQLException sqle) {
      throw new FileSystemException(sqle);
    }

    return in;
  }

  public void delete(final String path) throws FileSystemException {

    try {
      dao.remove(path(path, false));
    } catch (final SQLException sqle) {
      throw new FileSystemException(sqle);
    }
  }

  public void move(final String from, final String to) throws FileSystemException {

    try {
      dao.move(path(from, false), path(to, true));
    } catch (final SQLException | IOException ex) {
      throw new FileSystemException(ex);
    }
  }

  public void copy(final String from, final String to) throws FileSystemException {

    try {
      dao.copy(path(from, false), path(to, true));
    } catch (final SQLException | IOException ex) {
      throw new FileSystemException(ex);
    }
  }

  private int[] path(final String s, final boolean create) throws SQLException {

    String[] names = s.split(SEPERATOR);
    int[] path = range(rootPath, 0, rootPath.length + (names.length > 0 ? names.length - 1 : 0));

    File f = dao.read(rootPath);

    for (int level = 1; level < names.length; level++) {

      File p = f;

      f = dao.child(names[level], range(path, 0, level));

      if (null != f) {

        path[level] = (f.getNv() - p.getNv()) / p.getSnv();

      } else if (null == f && create) {

        path[level] = 1;
        File lc = dao.lastChild(range(path, 0, level));
        if (null != lc) {
          path[level] = (lc.getSnv() - p.getNv()) / p.getSnv();
        }

        f = dao.write(names[level], range(path, 0, level + 1));
      }
    }

    return path;
  }
}

An example usage of the above high level API for creating directories, reading and writing files, and moving sub trees is below.

Example.java

package org.adrianwalker.continuedfractions.filesystem.example;

import java.sql.Connection;
import java.sql.DriverManager;
import org.adrianwalker.continuedfractions.filesystem.File;
import org.adrianwalker.continuedfractions.filesystem.FileSystem;
import org.adrianwalker.continuedfractions.filesystem.FilesDAO;
import static org.adrianwalker.continuedfractions.filesystem.Printer.print;

public class Example {

  private static final String DRIVER = "org.postgresql.Driver";
  private static final String URL = "jdbc:postgresql://localhost:5432/postgres";
  private static final String USERNAME = "postgres";
  private static final String PASSWORD = "postgres";

  public static void main(final String[] args) throws Exception {

    Class.forName(DRIVER);
    Connection connection = DriverManager.getConnection(URL, USERNAME, PASSWORD);
    connection.setAutoCommit(false);

    FilesDAO dao = new FilesDAO(connection);
    dao.clear();

    FileSystem fs = new FileSystem(dao, 1);

    System.out.println("Create directories\n");

    fs.create("/bin");
    fs.create("/dev");
    fs.create("/etc");
    fs.create("/home");
    fs.create("/sbin");
    fs.create("/usr");

    fs.create("/home/adrian");
    fs.create("/home/other");

    fs.create("/home/adrian/documents/text");
    fs.create("/home/adrian/documents/presentations");
    fs.create("/home/adrian/documents/spreadsheets");

    connection.commit();

    print(fs.tree("/"));

    System.out.println("\nWrite files\n");

    fs.write("/home/adrian/documents/text/test1.txt", "Hello");
    fs.write("/home/adrian/documents/text/test2.txt", "Database");
    fs.write("/home/adrian/documents/text/test3.txt", "File System");
    fs.write("/home/adrian/documents/text/test4.txt", "World!");

    connection.commit();

    print(fs.tree("/"));

    System.out.println("\nMove files\n");

    fs.move("/home/adrian/documents", "/home/other");

    connection.commit();

    print(fs.tree("/"));

    System.out.println("\nPrint files\n");

    for (File file : fs.list("/home/other/documents/text")) {
      System.out.println(fs.read("/home/other/documents/text/" + file.getName()));
    }
  }
}

The above example code produces the output:

Create directories

/
  /bin
  /dev
  /etc
  /home
    /adrian
      /documents
        /text
        /presentations
        /spreadsheets
    /other
  /sbin
  /usr

Write files

/
  /bin
  /dev
  /etc
  /home
    /adrian
      /documents
        /text
          /test1.txt
          /test2.txt
          /test3.txt
          /test4.txt
        /presentations
        /spreadsheets
    /other
  /sbin
  /usr

Move files

/
  /bin
  /dev
  /etc
  /home
    /adrian
    /other
      /documents
        /text
          /test1.txt
          /test2.txt
          /test3.txt
          /test4.txt
        /presentations
        /spreadsheets
  /sbin
  /usr

Print files

Hello
Database
File System
World!

Source Code


Rule 30

$
0
0

Wolfram's Rule 30 is proper interesting, isn't it?

Here is a Java implementation:

Wolfram.java

package org.adrianwalker.cellularautomation;

import java.awt.Color;
import java.awt.image.BufferedImage;
import java.io.File;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;
import javax.imageio.ImageIO;

public final class Wolfram {

  private static final int MAX_TIME = 500;
  private static final int MAX_SPACE = MAX_TIME * 2;
  private static final int WHITE = Color.WHITE.getRGB();
  private static final int BLACK = Color.BLACK.getRGB();
  private static final String FORMAT = "png";
  private static final String OUTPUT = "/var/tmp/output.png";

  // cell to RGB lookup
  private static final Map<Integer, Integer> CELL_RGB = new HashMap<>();
  static {
    CELL_RGB.put(0, WHITE);
    CELL_RGB.put(1, BLACK);
  }

  // RGB to cell lookup
  private static final Map<Integer, Integer> RGB_CELL = new HashMap<>();
  static {
    RGB_CELL.put(WHITE, 0);
    RGB_CELL.put(BLACK, 1);
  }

  // http://en.wikipedia.org/wiki/Rule_30
  //
  // current pattern          111 110 101 100 011 010 001 000
  // new state for center cell 0   0   0   1   1   1   1   0
  private static final Map<Integer, Integer> RULE_30 = new HashMap<>();
  static {
    RULE_30.put(Arrays.hashCode(new int[]{1, 1, 1}), 0);
    RULE_30.put(Arrays.hashCode(new int[]{1, 1, 0}), 0);
    RULE_30.put(Arrays.hashCode(new int[]{1, 0, 1}), 0);
    RULE_30.put(Arrays.hashCode(new int[]{1, 0, 0}), 1);
    RULE_30.put(Arrays.hashCode(new int[]{0, 1, 1}), 1);
    RULE_30.put(Arrays.hashCode(new int[]{0, 1, 0}), 1);
    RULE_30.put(Arrays.hashCode(new int[]{0, 0, 1}), 1);
    RULE_30.put(Arrays.hashCode(new int[]{0, 0, 0}), 0);
  }

  public static void main(final String[] args) throws Throwable {

    BufferedImage image = new BufferedImage(MAX_SPACE, MAX_TIME, BufferedImage.TYPE_INT_RGB);

    init(image);
    execute(image, RULE_30);

    ImageIO.write(image, FORMAT, new File(OUTPUT));
  }

  private static void init(final BufferedImage image) {

    int time;
    int space;
    for (time = 0; time < MAX_TIME; time++) {
      for (space = 0; space < MAX_SPACE; space++) {
        image.setRGB(space, time, WHITE);
      }
    }

    time = 0;
    space = MAX_SPACE / 2;
    image.setRGB(space, time, BLACK);
  }

  private static void execute(final BufferedImage image, final Map<Integer, Integer> rule) {

    for (int time = 1; time < MAX_TIME; time++) {
      for (int space = 1; space < MAX_SPACE - 1; space++) {

        int[] pattern = {
          RGB_CELL.get(image.getRGB(space - 1, time - 1)),
          RGB_CELL.get(image.getRGB(space, time - 1)),
          RGB_CELL.get(image.getRGB(space + 1, time - 1))
        };

        int cell = rule.get(Arrays.hashCode(pattern));
        image.setRGB(space, time, CELL_RGB.get(cell));
      }
    }
  }
}

Output


(Click To Enlarge)

Random Gutenberg

$
0
0

Random Gutenberg is a Twitter bot which tweets random sentences from random Project Gutenberg eBooks.

RandomGutenberg.java

package org.adrianwalker.randomgutenberg;

import java.io.BufferedReader;
import java.io.InputStreamReader;
import static java.lang.String.format;
import java.net.URL;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Iterator;
import java.util.List;
import java.util.Random;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.zip.ZipInputStream;
import org.apache.log4j.Logger;
import twitter4j.Status;
import twitter4j.Twitter;
import twitter4j.TwitterFactory;

public final class RandomGutenberg {

  private static final Logger LOGGER = Logger.getLogger(RandomGutenberg.class);
  private static final int MAX_OFFSET = 3934400;
  private static final String GUTENBERG_ROBOT_URL = "http://www.gutenberg.org/robot/harvest?offset=%s&filetypes[]=txt";
  private static final Pattern HREF_PATTERN = Pattern.compile("href=\"(http://.*)\"");
  private static final Pattern SENTENCE_ENDINGS_PATTERN = Pattern.compile("(?<=\\s?\"?[.!?]\"?\\s?)");
  private static final Pattern WORD_ENDINGS_PATTERN = Pattern.compile("\\s");
  private static final int MAX_TRIES = 3;
  private static final int MAX_TWEET_LENGTH = 140;
  private static final int MIN_WORD_COUNT = 3;

  public static void main(final String[] args) {

    Random randomNumberGenerator = new Random(new Random(System.currentTimeMillis()).nextLong());

    String eBookText = null;

    for (int tries = 0; tries < MAX_TRIES; tries++) {
      try {
        eBookText = getRandomEbookText(randomNumberGenerator);
        break;
      } catch (final Throwable t) {
        LOGGER.error("Error getting eBook text", t);
      }
    }

    if (null == eBookText) {
      return;
    }

    String sentence = getRandomSentence(eBookText, randomNumberGenerator);

    try {
      tweet(sentence);
    } catch (final Throwable t) {
      LOGGER.error("Error sending tweet", t);
    }
  }

  private static String getRandomSentence(final String eBookText, final Random randomNumberGenerator) {

    List<String> sentences = new ArrayList<>(Arrays.asList(SENTENCE_ENDINGS_PATTERN.split(eBookText)));

    Iterator<String> sentenceIterator = sentences.iterator();
    while (sentenceIterator.hasNext()) {

      String sentence = sentenceIterator.next();
      String[] words = WORD_ENDINGS_PATTERN.split(sentence);

      int sentenceLength = sentence.length();
      int wordCount = words.length;

      if (wordCount < MIN_WORD_COUNT || sentenceLength > MAX_TWEET_LENGTH) {
        sentenceIterator.remove();
      }
    }

    String sentence = sentences.get(randomNumberGenerator.nextInt(sentences.size()));

    return sentence;
  }

  private static String getRandomEbookText(final Random randomNumberGenerator) throws Throwable {

    int offset = randomNumberGenerator.nextInt(MAX_OFFSET);
    URL url = new URL(format(GUTENBERG_ROBOT_URL, offset));

    BufferedReader reader = new BufferedReader(new InputStreamReader(url.openStream()));
    List<String> hrefs = new ArrayList<>();

    String line;
    while (null != (line = reader.readLine())) {

      Matcher matcher = HREF_PATTERN.matcher(line);

      if (matcher.find()) {
        hrefs.add(matcher.group(1));
      }
    }

    reader.close();

    String randomHref = hrefs.get(randomNumberGenerator.nextInt(hrefs.size()));

    url = new URL(randomHref);

    ZipInputStream zis = new ZipInputStream(url.openStream());
    zis.getNextEntry();
    reader = new BufferedReader(new InputStreamReader(zis));
    StringBuilder eBookBuffer = new StringBuilder();

    while (null != (line = reader.readLine())) {

      if (eBookBuffer.length() > 0) {
        eBookBuffer.append("");
      }

      line = line.trim();
      eBookBuffer.append(line);
    }

    reader.close();

    String eBookText = eBookBuffer.toString();

    return eBookText;
  }

  private static Status tweet(final String sentence) throws Throwable {

    String message = sentence;

    Twitter twitter = TwitterFactory.getSingleton();
    Status status = twitter.updateStatus(message);

    return status;
  }
}

Source Code

Desktop Upgrade – Part 1

$
0
0

I've not upgraded my desktop hardware for nearly seven years, so it's time for some new kit. To know how much extra bang for my buck I'm getting, I want to do a direct comparison between my current hardware and my new hardware, with the new kit running at stock speeds and also overclocked.

Current Hardware

My current hardware looks like this:

CPUIntel Core 2 Duo E8400
CoolerStock
RAMCorsair 4GB DDR2 800MHz
MotherboardASUS P5KC ATX
GraphicsInno3D GeForce 9500GT 1GB GDDR2

Benchmark

I used GeekBench to profile CPU performance and GpuTest for graphics, downloadable here:

http://www.primatelabs.com/geekbench/download
http://www.geeks3d.com/gputest/download/

I ran GeekBench three times to make sure the results were consistent:

RunSingle-Core ScoreMulti-Core ScoreFull Results
1st12723242http://browser.primatelabs.com/geekbench3/2019865
2nd12803244http://browser.primatelabs.com/geekbench3/2019892
3rd13173244http://browser.primatelabs.com/geekbench3/2019918

I ran each of the GpuTest fullscreen benchmark scripts, a couple couldn't run, the results of the ones which could are as follows:

ModulePointsFPS
Triangle20371339
Plot3D212735
FurMark1412
PixMark Volplosion540

Buy Stuff!

Now I'm thoroughly disgusted at the performance of my current rig it's time to buy some new stuff. I don't have stacks of cash to throw at this upgrade so I want to maximize performance per pound, easily overclock, and have further upgrade options in the future.

My new hardware will look like this:

CPUIntel Pentium Processor G3258£47.57
CoolerCooler Master Hyper 212 EVO£25.50
RAMHyperX FURY Series 8GB DDR3 1866MHz£52.99
MotherboardASRock Z97 Anniversary£54.00
GraphicsIntegrated Intel HD Graphicsn/a
 
Total£180.06

Not a bad price if this upgrade lasts another seven years.

There are some great prices on G3258 bundles out there, with the CPU already overclocked and ready to go, but they weren’t quite what I wanted, and I fancied clocking this myself. Some bundles I looked at are:

eclipsecomputers.com
dabs.com
overclockers.co.uk

Part 2

Desktop Upgrade – Part 2

Desktop Upgrade – Part 2

$
0
0

My new hardware came and it looks a bit like this:

Looking sexy

First things first - time to rag out my old kit and clean out seven years of accumulated dust.

Before: I wish my girlfriend was this dirty (that's not really my PC).

After: Loads cleaner than your mucky mother.

Brilliant, time to put it all together, that heat sink is an absolute beast and it was a bit of a sod to bolt to the motherboard. The Asrock Z97 Anniversary motherboard is only about 2/3 the size of my old Asus so there is a bit more room for messing about with the drives.

Assembled: The heatsink fits in the Cooler Master Centurion C5 case no problem if you remove the pipe attached to the case side.

After a fresh install of Linux Mint 17.1 with the Xfce desktop, we're ready to rock. The Linux 'sensors' command show that things are running pretty cool at stock speeds:

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0:  +27.0°C  (high = +80.0°C, crit = +100.0°C)
Core 0:         +27.0°C  (high = +80.0°C, crit = +100.0°C)
Core 1:         +26.0°C  (high = +80.0°C, crit = +100.0°C)

Testing

Before attempting any overclocking I wanted to make sure the new hardware was stable enough and cool enough with the out of the box configuration.

First I used Memtest86 to make sure that there were no faults with the memory, downloadable here:
http://www.memtest86.com/download.htm

Faultless

Memtest86 took about half an hour to run on 8GB RAM, and reported no errors after one pass. I couldn't really be bothered running it for more time - I was going to overclock it whatever the result.

Booting back into Linux, I installed Prime95 and ran the 'Small FTTs' torture test to bring the CPU up to full utilization. Prime95 is downloadable from here:
http://www.mersenne.org/download/

The 'sensors' command showed the temperature immediately go up to the low 40°C range, still pretty cool, and increase slow by about 1°C per hour:

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0:  +42.0°C  (high = +80.0°C, crit = +100.0°C)
Core 0:         +42.0°C  (high = +80.0°C, crit = +100.0°C)
Core 1:         +38.0°C  (high = +80.0°C, crit = +100.0°C)

+1 hour

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0:  +43.0°C  (high = +80.0°C, crit = +100.0°C)
Core 0:         +43.0°C  (high = +80.0°C, crit = +100.0°C)
Core 1:         +39.0°C  (high = +80.0°C, crit = +100.0°C)

+2 hours

ccoretemp-isa-0000
Adapter: ISA adapter
Physical id 0:  +44.0°C  (high = +80.0°C, crit = +100.0°C)
Core 0:         +44.0°C  (high = +80.0°C, crit = +100.0°C)
Core 1:         +40.0°C  (high = +80.0°C, crit = +100.0°C)

No errors or warning were reported during the tests:

[Worker #2 Mar 7 21:21] Torture Test completed 182 tests in 2 hours, 23 minutes - 0 errors, 0 warnings.
[Worker #2 Mar 7 21:21] Worker stopped.
[Worker #1 Mar 7 21:21] Torture Test completed 181 tests in 2 hours, 23 minutes - 0 errors, 0 warnings.
[Worker #1 Mar 7 21:21] Worker stopped.
[Main thread Mar 7 21:21] Execution halted.

Stock Benchmark

I benchmarked the new setup at stock speeds using the same tools and tests as used in part 1.

I ran GeekBench three times to make sure the results were consistent:

RunSingle-Core ScoreMulti-Core ScoreFull Results
1st28715180http://browser.primatelabs.com/geekbench3/2048335
2nd28695178http://browser.primatelabs.com/geekbench3/2048351
3rd28695178http://browser.primatelabs.com/geekbench3/2048367

I ran each of the GpuTest fullscreen benchmark scripts, as before:

ModulePointsFPS
Triangle39818663
Plot3D371861
FurMark3946
PixMark Volplosion1762

Compared to the benchmark from part 1, my new system is roughly twice as fast:

TestNumber of times faster
Single-Core Score2.2x
Multi-Core Score1.6x
Triangle2.0x
Plot3D1.7x
FurMark2.8x
PixMark Volplosion3.2x

Part 3

Desktop Upgrade – Part 3

Desktop Upgrade – Part 3

$
0
0

With my hardware tested and running well at stock speeds in part 1 and part 2, it's time to try for an overclock.

The Asrock Z97 motherboard detects the G3258 on boot up and prompts you to press the P key to enable 'Pentium Anniversary Boot'. Really this is just some pre-configured settings to get you started overclocking. Pressing P will gives you the following screen with some clock speeds to choose from:

I like big boosts and I can not lie

Obviously, when presented with this screen, you're just going to nail it straight on to the 4.2GHz option and see what happens.

Testing

I fired Prime95 up again with the same 'Small FFTs' torture as in part 2 and monitored temperatures using the Linux command 'sensors'. The temperatures reported were about 20°C hotter than before, but held absolutely solidy at the 60°C mark for two hours:

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0:  +60.0°C  (high = +80.0°C, crit = +100.0°C)
Core 0:         +60.0°C  (high = +80.0°C, crit = +100.0°C)
Core 1:         +52.0°C  (high = +80.0°C, crit = +100.0°C)

+1 hour

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0:  +60.0°C  (high = +80.0°C, crit = +100.0°C)
Core 0:         +60.0°C  (high = +80.0°C, crit = +100.0°C)
Core 1:         +51.0°C  (high = +80.0°C, crit = +100.0°C)

+2 hours

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0:  +60.0°C  (high = +80.0°C, crit = +100.0°C)
Core 0:         +60.0°C  (high = +80.0°C, crit = +100.0°C)
Core 1:         +51.0°C  (high = +80.0°C, crit = +100.0°C)

No errors or warning were reported during the tests:

[Worker #2 Mar 8 22:03] Worker stopped.
[Worker #1 Mar 8 22:03] Torture Test completed 177 tests in 1 hour, 43 minutes - 0 errors, 0 warnings.
[Worker #1 Mar 8 22:03] Worker stopped.
[Main thread Mar 8 22:03] Execution halted.

Overclocked Benchmark

I benchmarked the new setup at 4.2GHz using the same tools and tests as used in part 1 and part 2.

I ran GeekBench three times to make sure the results were consistent:

RunSingle-Core ScoreMulti-Core ScoreFull Results
1st35956562http://browser.primatelabs.com/geekbench3/2054643
2nd36006567http://browser.primatelabs.com/geekbench3/2054658
3rd35916558http://browser.primatelabs.com/geekbench3/2048367

Disappointingly the GPU benchmarks with the 4.2GHz 'Pentium Anniversary Boot' configuration came out exactly the same as at stock speeds in part 2, so I wont include them here. I suspect some custom settings tuning may yield better results.

Compared to the benchmark from part 1, my overclocked system is almost three times as fast on single core performance and twice as fast using multiple cores:

TestNumber of times faster
Single-Core Score2.8x
Multi-Core Score2.0x

Tweaking

The BIOS comes with loads of options for OC tweaking:

Tweak my MIPS harder

The core voltage looks a little high on the pre-configured overclock settings, so I've reduced that slightly, and I need to optimize settings for GPU and RAM, but for now I'm happy with my 4.2GHz clock.

There are some good tutorials out there for OCing this kind of rig. In the future I'll give a this guide a go and post the results here.

Viewing all 65 articles
Browse latest View live